Replacing environment variables in a properties file

In Linux, say I have the following file (e.g. conf.properties):

HOST_URL=http://$HOSTNAME
STD_CONFIG=http://$HOSTNAME/config
USER_CONFIG=http://$HOSTNAME/config/$unconfigured

I want to create another file with all the environment variables replaced...e.g. say the environment variable $HOSTNAME is 'myhost' and $unconfigured is not set, a script should produce the following output:

HOST_URL=http://myhost
STD_CONFIG=http://myhost/config
USER_CONFIG=http://myhost/config/

I was thinking this could be done in a sim开发者_运维技巧ple one-liner with some sort of sed/awk magic, but I'm no expert and my searches have been in vein, so appreciate any help.

Edit:

I should mention that the file can really be any format text file, for example xml. I just want to replace anything that looks like an env variable with whatever is currently set in the environment.

This is what envsubst is for.

echo 'Hello $USER'
Hello $USER
echo 'Hello $USER' | envsubst
Hello malvineous

You would probably use it more like this though:

envsubst < input.txt > output.txt

envsubst seems to be part of GNU gettext.

sed 's/$HOSTNAME/myhost/g;s/$unconfigured//g' yourfile.txt > another_file.txt

update:

Based on updates in your question, this won't be a good solution.

update2 :

This is based on an answer to a related question. I've hacked at it (I'm unfamiliar with perl) to remove undefined vars.

perl -p -e 's/\$\{([^}]+)\}/defined $ENV{$1} ? $ENV{$1} : $&/eg; s/\$\{([^}]+)\}//eg' yourfile.txt

Should work for any input text file, however you will need to define vars using the ${...} format which simplifies the string matching.

(rant regarding the evilness of eval moved to a separate post so as not to confuse readers)

"eval is evil"

This is not an answer, but a warning in response to using eval for this task. You really really really don't want to do that.

Exhibit 1: a malicious template file:

HOST_URL=http://$HOSTNAME
STD_CONFIG=http://$HOSTNAME/config
USER_CONFIG=http://$HOSTNAME/config/$unconfigured
&& cat /etc/redhat-release

An unsuspecting user:

[lsc@aphek]$ cat somefile | while read line; do echo $(eval echo `echo $line`); done
HOST_URL=http://xyz
STD_CONFIG=http://xyz/config
USER_CONFIG=http://xyz/config/
Red Hat Enterprise Linux WS release 4 (Nahant Update 9)

Note the last line!

Now, imagine the possibilities....

I'd do it like this:

# Set the $HOSTNAME and other variables
# Now evaluate the properties file as a shell script.
. config.properties
# Write the values
cat >somefile <<EOF
HOST_URL=$HOST_URL
STD_CONFIG=$STD_CONFIG
USER_CONFIG=$USER_CONFIG
EOF

Edit: Or this very nasty thing (I'm sure there's a better way)

for name in HOST_URL STD_CONFIG USER_CONFIG
    echo "$name=$(eval echo `echo '$'$name`)" >>somefile
end

Thanks to @DarkDust I came up with this:

cat somefile | while read line; do echo $(eval echo `echo $line`); done > somefile.replaced

I used this oneliner to replace ${VARIABLE} style variables in a file:

TARGET_FILE=/etc/apache2/apache2.conf; for VARNAME in $(grep -P -o -e '\$\{\S+\}' ${TARGET_FILE} | sed -e 's|^\${||g' -e 's|}$||g' | sort -u); do sed -i "s|\${$(echo $VARNAME)}|${!VARNAME}|g" ${TARGET_FILE}; done

I'm pretty sure someone can do this in 1/3rd of the length using awk… feel challenged! ;)

Here is a snippet of Javascript that I like to have around for solving this exact problem:

// A Javascript version of envsubst for our builds
// Purpose: replace all ocurrences of ${VAR} with the equivalent var from the environment from stdin
var readline = require('readline');

var rl = readline.createInterface({
  input: process.stdin,
  output: process.stdout,
  terminal: false
});

const environment = process.env;

rl.on('line', function(line) {
  const newLine = line.replace(/\$\{([a-zA-Z0-9_]+)\}/g, function(_match, variable) {
    const envVar = environment[variable];
    return envVar ? envVar : '';
  });

  process.stdout.write(`${newLine}\n`);
});

Hopefully this helps somebody else.

Here's a short one-liner that uses python's curly brace formatting to safely do the magic:

contents=\"\"\"`cat $file`\"\"\"; python -c "import os;print $contents.format(**os.environ)"

• avoids evil eval
• allows outputting curly braces: use {{ instead of {
• no need to specify vars explicitly when calling the script

For example, given properties file settings.properties:

# my properties file
someVar = {MY_ENV_VAR}
curlyBraceVar = has {{curly braces}}

Then, do the substitution with:

$ export MY_ENV_VAR="hello"
$ file=settings.properties 
$ contents=\"\"\"`cat $file`\"\"\"; python  -c "import os;print $contents.format(**os.environ)"
# my properties file
someVar = hello
curlyBraceVar = has {curly braces}

A script is here: https://raw.githubusercontent.com/aneilbaboo/machome/master/bin/substenv

if you have installed nodejs you can run

npx @utft/tt -e FOO=bar /path/to/input /path/to/output

or you can run it programmatically https://github.com/utftufutukgyftryidytftuv/tt