Bring a large pot of salted water to a rolling boil over high heat. Add the petite white or red potatoes and cook for about 15 minutes until fork-tender. Drain the potatoes in a colander and let them cool completely.

While the potatoes are cooling, grab a jar with a tight-fitting lid. Add extra-virgin olive oil, red wine vinegar, minced garlic, dry mustard, dried thyme, oregano, kosher salt, and black pepper to the jar. Seal the lid and vigorously shake it for about 30 seconds until well combined.

Once the potatoes are cool, cut them into bite-sized pieces. Transfer the chopped potatoes to a large mixing bowl and drizzle them with a bit of reserved caper brine. Let the potatoes absorb the briny goodness for about 10 minutes.

In the mixing bowl with the cooled potatoes, add pitted Kalamata olives, diced red onion, capers, oil-packed sun-dried tomatoes, and fresh dill. Gently toss the ingredients together.

Pour the prepared dressing over the potato mixture and carefully fold everything together using a large spoon or spatula. Let the Greek potato salad sit for at least 15 minutes to allow the flavors to meld.

Before serving, crumble fresh feta cheese over the top and gently mix it in. Adjust seasoning with more salt or pepper if desired. Serve at room temperature or chilled.