Wake On Lan Fails On Ubuntu

For my Dell XPS, the Wake On Lan feature always worked on Windows. I changed my desktop computer to Ubuntu and the Wake On Lan no longer works. The problem is actually in the driver that the maintainer apparently never fixes the WOL issue despite the many complaints. I found a fix for the driver but here is the script I was trying to use before the driver patching.

Usage

The script is called wol-manager.sh
The script, of course, has to be made executable like this…

chmod +x ./wol-manager.sh

The default for the script is to simply display an interactive terminal menu…

./wol-manager.sh
❌ Error: Use sudo.

sudo ./wol-manager.sh

--- WOL MANAGER ---
Select NICs to ENABLE WoL. Unselected will be DISABLED.
------------------------------------------------------------
Controls: [W/S] Up/Down, [SPACE] Toggle, [ENTER] Apply & Save
------------------------------------------------------------
    [X] ENABLE  lan   (Current: Enabled)
 >  [X] ENABLE  wifi  (Current: Disabled)
------------------------------------------------------------

Applying changes:
 [+] Enabling WoL on: lan
 [+] Enabling WoL on: wifi

✅ All changes applied successfully!

Because I typically just want all NICs enabled there is a parameter ‘-a’…

sudo ./wol-manager.sh -a

Applying changes:
 [+] Enabling WoL on: enp4s0
 [+] Enabling WoL on: wlp3s0

✅ All changes applied successfully!

Github Wol-Manager

If, like me, you have been applying for jobs, tailoring your resume multiple times to better fit the job description can get tiring. But aren’t we still unsure how effective it really is? I’m worried what questions the interviewer will ask about on my resume.

In today’s crazy competitive job market, making a resume that stands out is important. Tools like Ollama and Langchain offer nice useful AI models and tools that can not only be used to enhance your resume but also to prepare you for interviews.

Large Language Models (LLMs)

These are tools with which you may have interacted with through ChatGPT, Gemini, Claude or whatever. To use these to update a resume to get a better job we have to instruct, or prompt, them carefully. For me, it is best to make the prompt by role, task, input, and output method.

Define the role it is supposed to mimic, define its task in detail, tell what sort of inputs it should expect, and how it should give the results.

I’ve decided to take this a step further and develop my own application for my resumes.

Setup

For this application, I use the Llama3 model from Ollama with python and Langchain. Basically, this application will do these things:

• Read the job description provided
• Read your PDF resume
• Answer your questions based on your job description and resume
• Suggest modifications to your resume
• Do a mock interview

Instead of only working through the terminal, which I often do, I integrated streamlit into the AI Resume Chat application for an easier user interface.

Usage

The first entry, the job description, will be a part of the system prompt itself. It is because all of the conversations will be dependent on it.

The second entry, a PDF resume, is the next part of the system prompt. RAG gets us the required snippets from the uploaded resume and, based on these, the application can extract relevant parts of the previous chat history which may be required. All of this is provided to the LLM which gives us its insights.

The third entry, a drop down, allows the selection of a chat option:

• Enhanced Resume
• Simulate Interview Both options have their own prompt, one to help enhance a resume and one to conduct an interview based on the job description and the provided resume.

After clicking the Update button the model is loaded with all of the necessary provided information. When the model is ready a new entry appears when the user can ask any questions like “How can I change my resume for this job description?”, “What is wrong with my resume?”, or “Hello! Let’s start the interview please.”

If you want to use it then please visit the GitHub repository and apply the directions in the README.md file. All of this assumes you have knowledge of using python.

Github AI Resume Chat

Data Redaction

The term for removing private information from code is data redaction. This technique involves masking or obscuring sensitive data within a document or dataset to protect personally identifiable information (PII) and other confidential data. Additionally, the process of hiding or removing sensitive information from documents is referred to as document redaction.

Some Traditional Methods

Let’s say we have a docker compose for a complex set of services or maybe a python program for running an AI model. We want to share the code but, we don’t want to share our local information like the internal servers being used, their ports, the website domains, user names, passwords, contact information, or more.

I won’t bother explaining docker compose yaml files and their .env and secret files but they are the best solutions. For docker compose one would use the .env and secret files that are then placed in the .gitignore file so they don’t get deployed. The .env, though in the .gitignore file, can be replaced with an uploadable example like ‘example.env’ so the one using the code can see what they need to provide. The same applies for the secret files, the secret file can have an uploadable replacement like ‘example.db_root_password.txt’. Documentation is very important.

For an application, like for python, one might use environment variables and document what they should look like so the one using the code can know what they need to complete. A simple script to launch the python application can include those environment variables like this but excluding the launching script using .gitignore…

#!/bin/bash
# File: launch.sh

GPU_BASE_URL=”MyPc1.local”
GPU_BASE_PORT=”5005”

python ./my-ai-script.py

# File: .gitignore
.env
secrets/
launch.sh

My Custom Cover-All Redaction Method

I personally have been using a custom bash script that just goes through every folder and file in the repository replacing anything private with something else. The whole redaction process can be undone using the same script. All the private information is kept in a ‘private-information.txt’ file that is never in any repository. This way all the private information is in one place and before any commit, using a git pre-commit hook, everything gets redacted.

#!/bin/bash
# File: .git/hooks/pre-commit

count=$(/scripts/search.sh | wc -l)
if [ "$count" -gt 0 ]; then
  /scripts/replace.sh ../../
fi

The information used by search.sh and replace.sh is in the ‘private-information.txt’ file. All search and replace goes in the order of the longer string to be replaced down to the smaller string to be replaced so the longer one gets matched first.

One will notice that I am using an equal sign as a separator in this example. If one wants to use this script, just use a character not likely to be redacted like a whitespace or control character i.e. \r, \n, \v, etc

Here is an example of what the private-information.txt file would look like:

amigo.lan=MyPc1.local
hermano.lan=MyPc2.local
omigo=MyPc1
hermano=MyPc2
MySecretPassword1=<password>
MySecretPassword2=<Password>
MySecretPassword3=<Pwd>
MyUserName=<login>
me@mywebsite.net=user@domain.tld
contact@mywebsite.net=name@domain.tld
support@mywebsite.net=id@domain.tld
blog.mywebsite.net=sub.domain.tld
photos.mywebsite.net=immich.domain.tld
nc.mywebsite.net=nextcloud.domain.tld
mywebsite.net=domain.tld

Pretty simple, right? The whole point is to always cover anything that needs to be redacted with the ability to undo the redaction when needed. Let’s say one has all their repositories in one ‘repos’ folder. Every repository can then be kept redacted by running this script at any time.

What are these search and replace scripts I’m talking about? See ya in the next post!

Post: Search For Redacted Data

Data Redaction Search Script

One of the things I like to do when it comes to data redaction for my repositories is to check for any private information. I have this bash script called ‘search.sh’ that will search the whole folder for any private information. The script can do the opposite as well, searching for redacted information.

Please take a look at Data Redaction for the ‘private-information.txt’ that will be used here.

Sample run of search.sh

# Bad run...
/scripts/search.sh

Usage: /scripts/search.sh [option] <directory>
Options:
  -s | --swap          Search for redacted

# Search for private information
/scripts/search.sh ./

# Search for redacted information
/scripts/search.sh -s ./

For this script, since my repositories are always the same layout I define some bash variables that are easy to update. Please excuse the bad naming; they’ve been there for years.

# SECRETS is the file containing patterns to search for
SECRETS="/private-information.txt"
# EXD is a comma-separated list of directories to exclude
EXD="--exclude-dir={venv,.git,.vscode}"
# EXEXT is a comma-separated list of file extensions to exclude
EXEXT="--exclude=*.{svg,webp,png,jpg,pdf,docx,gz,zip,tar}"
# TEMPFILE is where to store the altered private information
TEMPFILE="/tmpfs/temp-patterns.txt"

Next, the script loads all the lines for the ‘private-information.txt’ file and puts them in the TEMPFILE location.

# Read in all the private information and put it all in the lines array
mapfile -t lines < $SECRETS

# For each line in the lines array we are going to create the actual
# file that will be used for searching later.
for line in "${lines[@]}"; do
  # In the example 'private-information.txt' each private and redacted
  # are separated by an equal sign
  IFS='=' read -r -a array <<< "$line"

  # Swap decides if we will use redacted or private
  if $swap; then
    echo "${array[1]}"
  else
    echo "${array[0]}"
  fi
done > $TEMPFILE
SECRETS="$TEMPFILE"

Now, we just run the grep command and see what is in there…

# There were problems with string interpolation so the script uses eval on a string
# $folder is the script parameter for the folder to search
command="grep $EXD $EXEXT -RiIFn -f $SECRETS $folder"
eval $command

The end result of running this script is either nothing or the file name, line number, and the text that matches the private or redacted information being searched for. As for replacing private with redacted; see ya in the next post!

Post: Replacing Redacted Data

Github Search.sh

Data Redaction Replace Script

It’s time to take a look at the script that will replace the private information with the redacted information. The same script can undo that switch of information making it very convenient when storing code on a remote repository without making it private. This bash script is called ‘replace.sh’ and it will search the whole folder for any private information replacing it with the redacted version.

Please take a look at Data Redaction for the ‘private-information.txt’ that will be used here.

Sample run of search.sh

# Bad run...
/scripts/replace.sh

Usage: /scripts/replace.sh [option] <directory>
Options:
  -s | --swap          Replace redacted with private

# Replace for private information with a redacted version
/scripts/replace.sh ./

# Replace the redacted version with original private information
/scripts/replace.sh -s ./

For this script, I’m adding a bash variable while the others are just like in search.sh. They are easy to update this way. Again, please excuse the bad naming; they’ve been there for years.

# SECRETS is the file containing patterns to search for
SECRETS="/private-information.txt"
# EXD is a comma-separated list of directories to exclude
EXD="--exclude-dir={venv,.git,.vscode}"
# EXEXT is a comma-separated list of file extensions to exclude
EXEXT="--exclude=*.{svg,webp,png,jpg,pdf,docx,gz,zip,tar}"
# SED_SCRIPT is the temporary sed script file
SED_SCRIPT="/tmpfs/script.sed"

This script uses the find command so we need to add the parameters to exclude the chosen directories and file extensions. I think it makes things easier to store the directories and file extensions to exclude in variables and then just generate what is needed for the find command.

This find command will be used like this…

find ./folder -type f \( ! -iname ".png" ! -iname ".zip" \) ! -path "./venv" ! -path "./.git" -print

To generate the directories to be excluded the following script is used:

# Build the exclude paths
path=""
IFS=',' read -r -a fld <<< "$EXD"
for d in "${fld[@]}"; do
  path="$path ! -path ""./${d}/*"""
done

To generate the file extensions to be excluded the following script is then used:

# Build the exclude extensions
extension="\( "
IFS=',' read -r -a ext <<< "$EXEXT"
for e in "${ext[@]}"; do
  extension="$extension ! -iname "".${e}"""
done
extension+=" \)"

This script also is going to use the sed command. Using the sed script file is best in this situation but the file has to be generated. The sed command will make changes directly to the target file while making a backup before making that change.

The sed command will be used like this…

# SED script file is, per line, like this...
# s/SEARCH/REPLACE/g

sed --in-place=.bak --file=/tmpfs/script.sed this-file.txt

To generate the SED script file we have to read the ‘private-information.txt’ and transform it taking into consideration the swap parameter. SED does not do non-regex substitutions so there will be escaping added for both private and redacted.

# Build the sed script from the secrets file
mapfile -t lines < $SECRETS

for line in "${lines[@]}"; do
  IFS='=' read -r -a array <<< "$line"

  # Swapped or not all entries must be escaped
  if $swap; then
    escaped_lhs="${array[1]//[][\\\/.^\$*]/\\&}"
    escaped_rhs="${array[0]//[\\\/&$'\n']/\\&}"
  else
    escaped_lhs="${array[0]//[][\\\/.^\$*]/\\&}"
    escaped_rhs="${array[1]//[\\\/&$'\n']/\\&}"
  fi
  escaped_rhs="${escaped_rhs%\\$'\n'}"

  echo "s/${escaped_lhs}/${escaped_rhs}/g"
done > $SED_SCRIPT

Next, we just need to get the list of files that can be modified…

# Find files, filter for ASCII, and create an array of files
command="find $folder -type f $extension $path -print"
results=$(eval $command | xargs file | grep ASCII | cut -d: -f1)
readarray -t files <<< "$results"

With the list of files that can be modified we will now apply the SED Script file to each of them.

for file in "${files[@]}"; do
   sed --in-place=.bku --file="$SED_SCRIPT" "$file"

   # Check if the file was actually modified by sed
   if ! cmp -s "$file" "$file.bak"; then
      # State which file was modified by sed
      echo "Modified $file"
   fi

   # Remove the backup copies
   rm "$file.bak"
done

And that is it. I put both search.sh and replace.sh in a git pre-commit hook. These files only need to be changed when there is a new type of folder or file type to ignore when replacing private and redacted information.

Github Replace.sh

Again, Please keep in mind, this particular job had a specific rule of not allowing any third-party libraries. Everything had to be done adhoc.

The Functions Keeps Failing Challenge

Imagine sending an email notification to a management group while making the server wait for the response from the email service. It didn’t cost much but I showed that it could be done using async without await. The server could respond to the client right away without waiting to hear back from the email service. The client shouldn’t really even know about the email being sent. The question from the team was, “What if the email service was down?” Well, we can log that error without telling the client. We could also try sending the email again. That is where this async retry function came into being.

Async Retry Function

/// <summary>
/// Asynchronously retry a failed function
/// </summary>
/// <param name="fun">Function to execute</param>
/// <param name="RetryTimes">Number of times to retry. Default is 5.</param>
/// <param name="delayMs">Delay between retries. Default is 500ms.</param>
static async Task<T> Retry<T>(Func<Task<T>> fun, int RetryTimes = 5, int WaitTime = 500)
{
    for (int i = 0; i < RetryTimes - 1; i++)
    {
        try
        {
            return await fun();
        }
        catch (Exception Ex)
        {
            Console.WriteLine($"Retry {i + 1}: Getting Exception : {Ex.Message}");
            await Task.Delay(WaitTime);
        }
    }

    // Last try
    return await fun();
}

Example

static async Task<string> ThisMightFail()
{
    const string notAllowed = "hello";

    await Task.Delay(500);

    Console.Write("Say something: ");
    var input = Console.ReadLine();

    if (input.Contains(notAllowed, StringComparison.OrdinalIgnoreCase))
    {
        Console.WriteLine($"You can't say '{notAllowed}'!");
        throw new ArgumentException($"You can't say '{notAllowed}'!");
    }

    return input;
}

try
{
    // Try ThisMightFail
    //   retry 3 times
    //   delay 1 second between each retry
    var result = await Retry(ThisMightFail, 3, 1000);
    Console.WriteLine(result);
}
catch (Exception ex)
{
    Console.WriteLine("Boo!");
}

What It Looks Like If It Fails

Say something: hello!
You can't say 'hello!'!
Retry 1: Getting Exception : You can't say 'hello!'!
Say something: hello
You can't say 'hello'!
Retry 2: Getting Exception : You can't say 'hello'!
Say something: byehello
You can't say 'byehello'!
Boo!

What It Looks Like If It Succeeds

Say something: hello!
You can't say 'hello!'!
Retry 1: Getting Exception : You can't say 'hello!'!
Say something: Bye!
Bye!

Okay, what about parameters?

static async Task<string> ThisMightFail(string p1, int p2, double p3) {...}

// Anonymous function
var result = await Retry(async () => ThisMightFail("Hello", 42, 3.14), 3, 1000);

Github AsyncRetry.cs

Keeping in mind, this particular job had a specific rule of not allowing any third-party libraries. Everything had to be done adhoc.

The Action Keeps Failing Challenge

While working with the team, I often found that tasks would fail due to other problems outside our domain. Often this was due to the SQL server timing out which we couldn’t change. There were sometimes problems where service we were relying on was slow starting up; all of our services would sleep after a mere 5 minutes.

We could just keep popping up the error notification and make the user resubmit but even I didn’t like doing that myself. I came up with an idea for going ahead and retrying until there was a definite point to give up.

A Retry Function

/// <summary>
/// Retry a failed action
/// </summary>
/// <param name="action">Action to perform</param>
/// <param name="numberOfRetries">Number of retries</param>
/// <param name="delayMs">Delay between reties. Default is no delay.</param>
public static void RetryAction(Action action, int numberOfRetries, int delayMs = 0)
{
    Exception? exception = null;
    int retries = 0;

    while (retries < numberOfRetries)
    {
        try
        {
            action();
            return;
        }
        catch (Exception ex)
        {
            // Ignore error
            exception = ex;
            retries++;
        }

        if (delayMs > 0)
        {
            Task.Delay(delayMs).Wait();
        }
    }

    throw exception!;
}

Example

public static void ThisMightFail()
{
    const int notAllowed = 1;

    Console.Write("Enter a number: ");
    var input = int.Parse(Console.ReadLine() ?? "0");

    if (input == notAllowed)
    {
        Console.WriteLine($"You number must not be {notAllowed}");
        throw new ArgumentException($"You number must not be {notAllowed}");
    }

    Console.Write("Number accepted!");
}

RetryAction(ThisMightFail /* The action to retry */,
            5 /* 5 retries */,
            500 /* 1/2 second delay */);

What It Looks Like If It Fails

Enter a number: 1
Your number must not be 1
Enter a number: 1
Your number must not be 1
Enter a number: 1
Your number must not be 1
Enter a number: 1
Your number must not be 1
Enter a number: 1
Message: Your number must not be 1
Source: RetryActionDemo
HelpLink:
StackTrace: at RetryActionDemo.Program.Main(String[] args) in ....

What It Looks Like If It Succeeds

Enter a number: 1
Your number must not be 1
Enter a number: 1
Your number must not be 1
Enter a number: 1
Your number must not be 1
Enter a number: 1
Your number must not be 1
Enter a number: 2
Number accepted!

Okay, what about parameters and return values…

public static int GetTheNumber(int notAllowed = 1)
{
    Console.Write("Enter a number: ");
    var input = int.Parse(Console.ReadLine() ?? "0");

    if (input == notAllowed)
    {
        Console.WriteLine($"You number must not be {notAllowed}");
        throw new ArgumentException($"You number must not be {notAllowed}");
    }

    Console.Write("Number accepted!");
    return input;
}

/* This is a wrapper so we can pass a parameter */
public static int ThisMightFail2()
{
    return GetTheNumber(3);
}

var res = RetryAction(ThisMightFail2 /* The action to retry */,
                      5 /* 5 retries */,
                      500 /* 1/2 second delay */);
Console.WriteLine(res);

The New RetryAction

/// <summary>
/// Retry a failed function
/// </summary>
/// <param name="fn">Function to perform</param>
/// <param name="numberOfRetries">Number of retries</param>
/// <param name="delayMs">Delay between reties. Default is no delay.</param>
public static TResult? RetryAction<TResult>(Func<TResult> fn, int numberOfRetries, int delayMs = 0)
{
    Exception? exception = null;
    int retries = 0;

    while (retries < numberOfRetries)
    {
        try
        {
            return fn();
        }
        catch (Exception ex)
        {
            // Ignore error
            exception = ex;
            retries++;
        }

        if (delayMs > 0)
        {
            Task.Delay(delayMs).Wait();
        }
    }

    throw exception!;
}

What It Looks Like If It Fails

Enter a number: 3
Your number must not be 3
Enter a number: 3
Your number must not be 3
Enter a number: 3
Your number must not be 3
Enter a number: 3
Your number must not be 3
Enter a number: 3
Message: Your number must not be 3
Source: RetryActionDemo
HelpLink:
StackTrace: at RetryActionDemo.Program.Main(String[] args) in ....

What It Looks Like If It Succeeds

Enter a number: 3
Your number must not be 3
Enter a number: 3
Your number must not be 3
Enter a number: 3
Your number must not be 3
Enter a number: 3
Your number must not be 3
Enter a number: 1
Number accepted!
1

Github RetryAction.cs

I’m just sharing experiences

I’ve worked with many teams and my role was mostly as a helper and problem solver. This is what I actually enjoy about my work. Well, that and ETL.

In a new job, I discovered that there was no indication on the server as to what stage of deployment the project was in. Yes, each deployment went to a different server based on the git branch but there was nothing on the server saying what the environment was. This meant I couldn’t use the typical variety of AppSettings configurations. I found that even web.config was blocked on deployment. I asked if we could fix this and the CI/CD pipeline manager said he had no idea how to do that or why it would even be needed. He even advised just modifying the program.cs for each deployment.

Why is this environment variable important?

So, why would this matter? Well, environment variables, like EnvironmentName, can decide what the server environment is going to be like. This can also be used to determine what actions to take in that environment or even provide secret information only the server admin knows.

Like many already do, we should have had different AppSettings configurations to configure each application for it’s current environment. The AppSettings configuration can include things like the application version, what database to use for that stage, where to send the “error” warning email to, where to send the automated application emails, who is to be the initial administrator, where to store files locally, etc. Basically, up until I found a workaround, we had put everything into one appsettings.json for all stages and the project.cs code had to be modified for each deployment. What could go wrong with that?

My AppSettings suggestions were based on Microsoft’s Learning website…

• appsettings.development.json - A developers environment would be radically different as the developer would be using a different database (locally), they might want to disable the application emails being sent out, who the application admin was (the developer obviously), where to store the local files, and more.

• appsettings.stage.json – The stage environment would be nothing like the developers environment but it should work just like the producation environment. It would of course have it’s own database, some emails might be different, and storing the files would be in a different place on that server.

• appsettings.production.json – The production environment would be like the stage environment only far more stable. The application server and the database servers should have been faster and more stable as well. Experiments should not have been happening in this environment because they would throw the user’s into appropriate tantrums.

One of these AppSettings versions would be chosen based on the EnvironmentName environment variable.

My project hack

I made a fun change to each of the projects so the proper AppSettings configuration would be used on deployment. It did take the team a while to adjust but the code got a lot cleaner and the deployments were more stable.

Here is the MSBuild modification I made:

Something.csproj

  <PropertyGroup>
    <GitBranch>
       $([System.IO.File]::ReadAllText('$(MSBuildThisFileDirectory)..\..\.git\HEAD').Replace('ref: refs/heads/', '').Trim())
    </GitBranch>
  </PropertyGroup>

   <Choose>
      <WhenCondition="'$(GitBranch)'=='main'And'$(Configuration)'=='Release'">
         <PropertyGroup>
            <EnvironmentName>Production</EnvironmentName>
         </PropertyGroup>
      </WhenCondition=>
      <WhenCondition="'$(GitBranch)'=='develop'And'$(Configuration)'=='Release'">
         <PropertyGroup>
            <EnvironmentName>Staging</EnvironmentName>
         </PropertyGroup>
      </WhenCondition=>
      <Otherwise>
         <PropertyGroup>
            <EnvironmentName>Development</EnvironmentName>
         </PropertyGroup>
      </Otherwise>
   </Choose>

What is happening here is that, in GitBranch, I’m having MSBuild look for the .git folder and then read the HEAD file. Using the String.Replace() function, ‘ref: refs/heads/’ is removed leaving just the branch name. The branch name is trimmed and then stored in the GitBranch custom property.

Choose is a feature MSBuild supports in csproj files and it lets the developer make a choice based on the property being used. In this project, ‘main’ was used for production, ‘develop’ was used for stage, and thus anything not ‘Release’ would be a ‘Development’ environment. The WhenCondition simply evaluates GitBranch and Configuration as strings and compares them to the conditions that should be met. When chosen, the EnvironmentName environment variable gets set. It is basically like a C# switch statement or expression.

Environment.SetEnvironmentVariable("EnvironmentName",
  (GitBranch, Configuration) switch
  {
    ("main", "Release") => "Production",
    ("develop", "Release") => "Staging",
    _ => "Development"
  });
}

While the rest of the team were getting paid more than I was, I admit that I was hired to help them fix problems and help migrate old applications. I love helping and I’m not criticizing the team here, though it might sound like it. I like helping a team that listens.

Github Repo

NameCheap

Some of my domain names are registered with NameCheap. Like my other registrars, I often use DDNS but NameCheap is a lot more difficult to keep the IP addresses up to date. Rather than use their provided application I made my own a long time ago because it was a fun challenge but I’ve decided to share my DDNS IP address updating script.

As a note, I also have been using an Asus router for many years now. They provide their own DDNS service with their own provider domain named ‘asuscomm.com’. It does come with it’s own SSL certificate also. You get to choose your own subdomain. As an example, mine could be ‘asus-ddns-subdomain’ giving me a final url like ‘asus-ddns-subdomain.asuscomm.com’. My ‘asus-ddns-subdomain.asuscomm.com’ url has the IP address assigned by my internet provider and Asus keeps it up to date. This is so very useful in a huge variety of ways which is one of several reasons I still use an Asus router.

DDNS.sh Script

The script I’m using is named ddns.sh and I keep it in my docker folder at /svr/docker so it is easy to find and edit when needed. This script will require the use of CRON but it will mostly only be run very quickly without sending any data back to NameCheap unless it becomes necessary.

Disable Echo

First I wanted to be able to turn echo off when it isn’t needed. I’m using the environment variable CRON as a flag to turn echo off. For echo, I’m using a variable that starts with the echo command and, if the CRON environment variable is there, the variable changes to “:” which just means “don’t do anything”.

echo="echo"
if [ ! -z "$CRON" ]; then
  echo=":"
fi

Domains and Subdomains

The next step is to create two arrays, one for the root domains and the other for the subdomains. This has actually made it pretty easy to add new domains or subdomains and the CRON job doesn’t even have to be restarted.

Each array index is the root domain. On NameCheap, every domain has it’s own API Access Key for updating the IP address which is why the need for them to be included here.

Every subdomain, indexed by their own root domain, is also provided here in the script. For NameCheap, the subdomains don’t know the root domain’s IP address so they need to be set individually. Each subdomain here is separated by a ‘|’.

Please note, every subdomains element starts with ‘@’. This is the reference to the root domain itself. If you are using this script and have a domain that doesn’t have any subdomains, like ‘domain6.tld’, then you still need to provide the API key, of course, and the subdomains entry must, at the very least, have the ‘@’ listed and, in this case, no ‘|’ will be needed.

declare -A domains
declare -A subdomains

domains["domain4.tld"]="random-namecheap-api-key-one"
domains["domain5.tld"]="random-namecheap-api-key-two"
domains["domain3.tld"]="random-namecheap-api-key-three"
domains["domain6.tld"]="random-namecheap-api-key-four"

subdomains["domain4.tld"]="@|www.|immich.|nextcloud.|collabora."
subdomains["domain5.tld"]="@|www."
subdomains["domain3.tld"]="@|www."
subdomains["domain6.tld"]="@"

IP Address Found

From here, the script needs to know what the currently assigned IP address is. Well, the script will be using the Asus provided url. The host command gives us a break down of the url provided. Grep find the ‘has address’ part. Awk extracts the IP address. The IP address is then assigned to the local_ip variable.

ddns="asus-ddns-subdomain.asuscomm.com"
local_ip=$(host -t a $ddns | grep "has address" | awk '{print $4}')
$echo "Local IP: $local_ip"

Iterating Over The Domains

We need to report the new IP address to NameCheap for every root domain. The ‘!’ is saying to return the keys of each element of the array. The script will go through every root domain like this…

for domain in "${!domains[@]}"; do
...
done

Iterating Over Subdomains

For each root domain we also need to report the new IP address to NameCheap for every subdomain. The script continues by getting the subdomains and splitting them up into an array called ‘hosts’ using the bash ‘declare -a’ operation. The script can then iterate over the subdomains even if it is just an ‘@’.

for domain in "${!domains[@]}"; do
  str=${subdomains[$domain]}
  IFS='|'
  declare -a hosts=($str)

  for host in "${hosts[@]}"; do
  ...
  done
done

Authoritative Name Servers

The script needs to determine the authoritative name server for each subdomain as we will be asking that name server what the current IP address is for that subdomain.

Here, if the subdomain, represented as host, is empty then it will be given and ‘@’ just in case. The actual host command is used to find the name server for the subdomain. The result has a lot of information so ‘grep’ looks for the name server part. Just in case there is more than one name server listed the head command gets the first one. The awk command then returns the name server’s name.

hst="${host//@/}"
authoritative_nameservers=$(host -t ns $hst$domain | grep "name server" | head -n1 | awk '{print $4}')

$echo "Authoritative nameserver for $hst$domain: $authoritative_nameservers"

Subdomain IP Address

This seems silly to do things this way when we could just use something like ‘host -t a $hst$domain’ but, in this situation, it is best to get the IP address from the authoritative name server. The dig command is the best way I could find at the time for this purpose.

resolved_ip=$(dig +short @$authoritative_nameservers $hst$domain)

$echo "Resolved IP for $hst$domain: $resolved_ip"

Do We Need To Anything?

At this point, we can compare the resolved IP address with the local IP address to determine if anything has changed. Regardless of the solution, the subdomain and domain loops continue to their end.

if [ "$resolved_ip" = "$local_ip" ]; then
  $echo "$domain records are up to date!"
else
  ...
fi

Update The NameCheap DDNS IP Address

If the resolved IP address does not match the local IP address then it is time to tell NameCheap DDNS that the IP address has changed for the current subdomain.

This is done by using their DDNS update API via the curl command. We provide the subdomain, or just an ‘@’ if needed, the root domain, and the “password” that is actually the API Access Key.

response=$(curl -s "https://dynamicdns.park-your-domain.com/update?host=${host//./}&domain=$domain&password=${domains[$domain]}")

Checking the response from the NameCheap API, after trying to update the IP address, for errors is a big help. This is best run independently outside of CRON so you can see any error message.

err_count=$(grep -oP "<ErrCount>\K.*(?=</ErrCount>)" <<<"$response")
err=$(grep -oP "<Err1>\K.*(?=</Err1>)" <<<"$response")

if [ "$err_count" = "0" ]; then
    $echo "API call successful! DNS propagation may take a few minutes..."
else
    echo "API call failed! Reason: $err"
fi

CRON for DDNS.sh Script

CRON will run the script at short intervals watching for my internet service provider changing my IP address since they don’t even tell me when it happens. I’m going to assume you know how to use CRON.

Running ‘crontab -e’ will prompt you to choose your editor if you haven’t used it before.

Add the following line at the end of the crontab list of tasks. The task runs the ddns.sh script every 10 minutes from the top of the hour. You are free to adjust that timing as you want. I’ve seen people using 5 minute intervals, 1 hour intervals, and others. Including ‘CRON=running’, as described above, adds the CRON environment variable to the script telling it to not use echo.

0,10,20,30,40,50 * * * * CRON=running /svc/docker/ddns.sh

Github ddns.sh

Most people are not aware that there are standards for JSON – or lots of other things in the software development world. Let’s have a look; if you are a learner this will push you forward…

JSON stands for "JavaScript Object Notation", a simple data interchange format. There are lots of these data interchange formats but JSON is great for web development due to its strong ties to Javascript and, of course, Java.

JSON Structure

JSON supports the following data structures but they all have different interpretations in various programming languages:

• object:

  { ‘key1’: ‘value1’, ‘key2’: ‘value2’ }

• array:

  [ ‘first’, ‘second’, ‘third’ ]

• number:

  42

  3.1415926

• string:

  ‘This is a string’

• boolean:

  true

  false

• null:

  null

What some people might expect with JSON:

{
  'name': 'John Doe',
  'birthday': 'February 22, 1978',
  'address': 'Richmond, Virginia, United States'
}

What JSON should look like:

{
  'first_name': 'John',
  'last_name': 'Doe',
  'birthday': '1978-02-22',
  'address': {
    'home': true,
    'street_address': '2020 Richmond Blvd.',
    'city': 'Richmond',
    'state': 'Virginia',
    'country': 'United States'
  }
}

One of the JSON sample representations is better than the other. The first is quick and dirty while the second has a thought out structure. A schema makes the second one even better and easier to interpret when one could need to support multiple consumers or using various programming languages.

This is an example of the JSON Schema for the second JSON sample. Note how it is still JSON also:

{
  '$id': 'https://yoursite.com/person.schema.json',
  '$schema': 'https://json-schema.org/draft/2020-12/schema',
  'title': 'Person',
  'type': 'object',
  'properties': {
    'first_name': { 'type': 'string' },
    'last_name': { 'type': 'string' },
    'birthday': { 'type': 'string', 'format': 'date' },
    'address': {
      'type': 'object',
      'properties': {
        'home': {'type': 'boolean' }
        'street_address': { 'type': 'string' },
        'city': { 'type': 'string' },
        'state': { 'type': 'string' },
        'country': { 'type' : 'string' }
      }
    }
  }
}

Declaring a JSON Schema

A schema needs to be shared. It is a way of saying "This is what we expect from our clients and what we will deliver." Doesn’t this make life simpler rather than guessing around until something works?

Is this something new? Nope, even XML had schema. Do you remember XSL? There was a reason for that. Every programming language out there has a "schema" as well.

In the above example JSON schema one should have noticed the "type" keyword. A JSON property can often be misinterpreted. "Is it a number? A string? An object? What do I provide?", would be some common questions.

In the schema, "type" specifies the JSON type that is accepted. Sometimes a producer / consumer can handle multiple types but we need to be specific.

{ 'type': 'number' }

Accepts number only.

 { 'type': [number, string] }

Accepts number or string.

{ 'type': 'integer' }

Accepts integer number only.

Additional attributes can accompany a "type". For example, one could add ‘description’, ‘minimum’, ‘maximum’, ‘minLength’, ‘maxLength’, ‘pattern’, ‘format’ and more properties. Just stick to the standards one makes.

The JSON schema also includes annotations like ‘$schema’ (points to the type of schema being used), ‘title’, ‘description’, ‘default’, ‘examples’, ‘$comment’, ‘enum’, ‘const’, ‘required’ and many more.

Non-JSON Data

To include non-JSON data one can also make use of the schema to clarify what is being passed around by using annotations like ‘contentMediaType’ and ‘contentEncoding’.

As an example, the proposed schema:

{
  'type': 'object',
  'properties': {
    'contentEncoding':  { 'type': 'string' },
    'contentMediaType':  { 'type': 'string' },
    'data':  { 'type': 'string' }
  }
}

The sample data:

{
  'contentEncoding': 'base64',
  'contentMediaType': 'image/png',
  'data': 'iVBORw0KGgoAAAANSUhEUgAAABgAAAA...'
}

References

For more details: https://json-schema.org

Specification: https://json-schema.org/specification.html

If you need help making a JSON schema: https://jsonschema.net/