xml - Unable to correctly use variable in curl command

Question

I am using a curl command and its using quite a lot of variables. And these variables contain XML.

My main XML file is:

<?xml version="1.0" encoding="UTF-8"?>
<md:EntityDescriptor ID="_16x6ttxhv1vzbzoop6nddfaece4r9ota2tem7as" entityID="http://jenkins-0.com:8080/securityRealm/finishLogin" validUntil="2038-10-24T23:16:50.279Z" xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata"><md:Extensions xmlns:alg="urn:oasis:names:tc:SAML:metadata:algsupport"><alg:DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha512"/><alg:DigestMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#sha384"/><alg:DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/><alg:DigestMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#sha224"/><alg:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/><alg:DigestMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha512"/><alg:DigestMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha384"/><alg:DigestMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"/><alg:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1"/><alg:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#dsa-sha1"/></md:Extensions><md:SPSSODescriptor AuthnRequestsSigned="true" WantAssertionsSigned="true" protocolSupportEnumeration="urn:oasis:names:tc:SAML:2.0:protocol urn:oasis:names:tc:SAML:1.0:protocol urn:oasis:names:tc:SAML:1.1:protocol"><md:Extensions xmlns:init="urn:oasis:names:tc:SAML:profiles:SSO:request-init"><init:RequestInitiator Binding="urn:oasis:names:tc:SAML:profiles:SSO:request-init" Location="http://jenkins-0.com:8080/securityRealm/finishLogin"/></md:Extensions><md:KeyDescriptor use="signing"><ds:KeyInfo xmlns:ds="http://www.w3.org/2000/09/xmldsig#"><ds:X509Data><ds:X509Certificate>MIIC+TCCAeGgAwIBAgIUARYMLNA/4YtY3NmaNBuptIz+NrkwDQYJKoZIhvcNAQELBQAwFzEVMBMG
A1UEAwwMU0FNTC1qZW5raW5zMB4XDTE4MTAyNDEzMzMxMFoXDTE5MTAyNDEzMzMxMFowFzEVMBMG
A1UEAwwMU0FNTC1qZW5raW5zMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAjzJjDox7
K5+bosd4aC6Kd5pLtrxTegNR5jGzRXajEMEqpP9ZkwprXXS6o2hcRa6kolPhuB5TxJ2xj+pBmVL+
cji9OW1xeQQWrduNxobLH2eT2FjABrM9kQCVUkf1EMndQRMN0ktScWSG3JGpCWe1Hz7RY3lFjiYy
U8kzTkCi6FRdrFj3gVZx3m82WHWZBku5KVfvl00+sRvjxCWASOiQ5GhJr4o2Q2cGKiESOk9Gs5nT
Po6AdQ5KiXT4Wrk/c4F0+YGIhQjU3ud85yqnBGWePjA5ZRPZ14/cqlB08byfftxvZmdgnfutGYqW
GQrBAx8yMX8wjVh1+ImxQGkd2Tw6swIDAQABoz0wOzAdBgNVHQ4EFgQUrlazwRKNnly9/VqoX6Sr
+jol304wGgYDVR0RBBMwEYIPY249U0FNTC1qZW5raW5zMA0GCSqGSIb3DQEBCwUAA4IBAQAMwzWL
KL4JCquNYXL7njBoiYCUFwytGsItTkqHWClrNMUGFKlyH+Te6srg4uAqSQsPwG9hQ3F/uBCWl1Fm
dTtHxpSTAYfMHceoEOfrRsdF8fazsYqZ1Un6pvxW++BS+Xzb2ljnw1OyeAJv8Bzjz9blaLntPTsX
epR0MGjWqL3YBQbGyyAVQPKGbAtA3iCmTwRrtMjsz2IUgQfWn5VzrowaPrH1G1v+IFUQLllQTDVW
Yaj8JbRrQ+RkgHa8e/6cG5VimTici6kp3/eQDwLPpPzSYwMXadlLZcWOKMEakDxoAw3YO4nb7VZY
P7FJZdF8Ptzb33LcD5iZC9MO3Dd1/DUZ</ds:X509Certificate></ds:X509Data></ds:KeyInfo></md:KeyDescriptor><md:KeyDescriptor use="encryption"><ds:KeyInfo xmlns:ds="http://www.w3.org/2000/09/xmldsig#"><ds:X509Data><ds:X509Certificate>MIIC+TCCAeGgAwIBAgIUARYMLNA/4YtY3NmaNBuptIz+NrkwDQYJKoZIhvcNAQELBQAwFzEVMBMG
A1UEAwwMU0FNTC1qZW5raW5zMB4XDTE4MTAyNDEzMzMxMFoXDTE5MTAyNDEzMzMxMFowFzEVMBMG
A1UEAwwMU0FNTC1qZW5raW5zMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAjzJjDox7
K5+bosd4aC6Kd5pLtrxTegNR5jGzRXajEMEqpP9ZkwprXXS6o2hcRa6kolPhuB5TxJ2xj+pBmVL+
cji9OW1xeQQWrduNxobLH2eT2FjABrM9kQxVUkf1EMndQRMN0ktScWSG3JGpCWe1Hz7RY3lFjiYy
U8kzTkCi6FRdrFj3gVZx3m82WHWZBku5KVfvl00+sRvjxCWASOiQ5GhJr4o2Q2cGKiESOk9Gs5nT
Po6AdQ5KiXT4Wrk/c4F0+YGIhQjU3ud85yqnBGWePjA5ZRPZ14/cqlB08byfftxvZmdgnfutGYqW
GQrBAx8yMX7wjVh1+ImxQGkd2Tw6swIDAQABoz0wOzAdBgNVHQ4EFgQUrlazwRKNnly9/VqoX6Sr
+jol304wGgYDVR0RBBMwEYIPY249U0FNTC1qZW5raW5zMA0GCSqGSIb3DQEBCwUAA4IBAQAMwzWL
KL4JCquNYXL7njBoiYCUFwytGsItTlqHWClrNMUGFKlyH+Te6srg4uAqSQsPwG9hQ3F/uBCWl1Fm
dTtgxpSTAYfMHceoEOfrRsdF8fazsYqZ1Un6pvxW++BS+Xzb2ljnw1OyeAJv8Bzjz9blaLntPTsX
epR0MGjWqL3YBQbGyyAVQPKGbAtA3iCmTwRrtMjsz2IUgQfWn5VzrowaPrH1G1v+IFUQLllQTDVW
Yaj8JbRrQ+RkgHa8e/6cG5VimTici6kp3/eQDwLPpPzSYwMXadlLZcWOKMEakDxoAw3YO4nb7VZY
P7FJZdF8Ptzb33LcD5iZC9MO3Dd1/DUZ</ds:X509Certificate></ds:X509Data></ds:KeyInfo></md:KeyDescriptor><md:NameIDFormat>urn:oasis:names:tc:SAML:2.0:nameid-format:transient</md:NameIDFormat><md:NameIDFormat>urn:oasis:names:tc:SAML:2.0:nameid-format:persistent</md:NameIDFormat><md:NameIDFormat>urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress</md:NameIDFormat><md:NameIDFormat>urn:oasis:names:tc:SAML:1.1:nameid-format:unspecified</md:NameIDFormat><md:AssertionConsumerService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" Location="http://jenkins-0.com:8080/securityRealm/finishLogin" index="0"/></md:SPSSODescriptor></md:EntityDescriptor>

I am doing some post processing i.e. removing new lines and escaping ". Im saving the result in a variable called POST_DATA

POST_DATA=$(cat PRE_DATA.xml | sed -e ':a' -e 'N' -e '$!ba' -e 's/
/\n/g' | sed 's/"/"/g')

Now I want to send this XML file as a variable to the --data field of my curl command.

curl -X PUT 
  https://my-end-point 
  -H 'Authorization: Basic <TOKEN>' 
  -H 'Content-Type: application/json' 
  -d '{
  "metadata": "$DATA",
  "id": "$SP_ID"
}'

This clearly did not work as I was not sending the variables correctly.

I tried the following :

curl -X PUT 
  https://my-end-point 
  -H 'Authorization: Basic <TOKEN>' 
  -H 'Content-Type: application/json' 
  -d "{
  "metadata": "${DATA}",
  "id": "${SP_ID}"
}"

I also tried this :

curl -X PUT 
  https://my-end-point 
  -H 'Authorization: Basic <TOKEN>' 
  -H 'Content-Type: application/json' 
  -d "$( printf '{ "metadata": "%s", "id": "%s" }' 
           "${DATA}" "${SP_ID}" )"

For all the above scenarios, I get:

Invalid JSON provided as input.

None of these solutions work. However when I use Postman app, it works perfectly fine. Basically I populate metadata with the actual value and it works. I replicated the curl command with the actual value in the metadata field and it works. But when I send that value to a variable and use that variable, it does not work. Ive exhausted all ideas on how to make this work. Any ideas ?

Answer 1

A simple way to escape arbitrary text in JSON is to use jq. Example session:

$ DATA='<?xml version="1.0" encoding="UTF-8"?>
> <md:EntityDescriptor'
$ SP_ID=1
$ jq --arg metadata "$DATA" --arg id "$SP_ID" '{"metadata": $metadata, "id": $id}' <<< {}
{
  "metadata": "<?xml version="1.0" encoding="UTF-8"?>
<md:EntityDescriptor",
  "id": "1"
}

The jq output can then be used in the curl command.