Use a database server running as a container
Tip
This content is an excerpt from the eBook, .NET Microservices Architecture for Containerized .NET Applications, available on .NET Docs or as a free downloadable PDF that can be read offline.
You can have your databases (SQL Server, PostgreSQL, MySQL, etc.) on regular standalone servers, in on-premises clusters, or in PaaS services in the cloud like Azure SQL DB. However, for development and test environments, having your databases running as containers is convenient, because you don't have any external dependency and simply running the docker-compose up command starts the whole application. Having those databases as containers is also great for integration tests, because the database is started in the container and is always populated with the same sample data, so tests can be more predictable.
SQL Server running as a container with a microservice-related database
In eShopOnContainers, there's a container named sqldata, as defined in the docker-compose.yml file, that runs a SQL Server for Linux instance with the SQL databases for all microservices that need one.
A key point in microservices is that each microservice owns its related data, so it should have its own database. However, the databases can be anywhere. In this case, they are all in the same container to keep Docker memory requirements as low as possible. Keep in mind that this is a good-enough solution for development and, perhaps, testing but not for production.
The SQL Server container in the sample application is configured with the following YAML code in the docker-compose.yml file, which is executed when you run docker-compose up. Note that the YAML code has consolidated configuration information from the generic docker-compose.yml file and the docker-compose.override.yml file. (Usually you would separate the environment settings from the base or static information related to the SQL Server image.)
sqldata:
image: mcr.microsoft.com/mssql/server:2017-latest
environment:
- SA_PASSWORD=Pass@word
- ACCEPT_EULA=Y
ports:
- "5434:1433"
In a similar way, instead of using docker-compose, the following docker run command can run that container:
docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=Pass@word' -p 5433:1433 -d mcr.microsoft.com/mssql/server:2017-latest
However, if you are deploying a multi-container application like eShopOnContainers, it is more convenient to use the docker-compose up command so that it deploys all the required containers for the application.
When you start this SQL Server container for the first time, the container initializes SQL Server with the password that you provide. Once SQL Server is running as a container, you can update the database by connecting through any regular SQL connection, such as from SQL Server Management Studio, Visual Studio, or C# code.
The eShopOnContainers application initializes each microservice database with sample data by seeding it with data on startup, as explained in the following section.
Having SQL Server running as a container is not just useful for a demo where you might not have access to an instance of SQL Server. As noted, it is also great for development and testing environments so that you can easily run integration tests starting from a clean SQL Server image and known data by seeding new sample data.
Additional resources
Run the SQL Server Docker image on Linux, Mac, or Windows
https://video2.skills-academy.com/sql/linux/sql-server-linux-setup-dockerConnect and query SQL Server on Linux with sqlcmd
https://video2.skills-academy.com/sql/linux/sql-server-linux-connect-and-query-sqlcmd
Seeding with test data on Web application startup
To add data to the database when the application starts up, you can add code like the following to the Main method in the Program class of the Web API project:
public static int Main(string[] args)
{
var configuration = GetConfiguration();
Log.Logger = CreateSerilogLogger(configuration);
try
{
Log.Information("Configuring web host ({ApplicationContext})...", AppName);
var host = CreateHostBuilder(configuration, args);
Log.Information("Applying migrations ({ApplicationContext})...", AppName);
host.MigrateDbContext<CatalogContext>((context, services) =>
{
var env = services.GetService<IWebHostEnvironment>();
var settings = services.GetService<IOptions<CatalogSettings>>();
var logger = services.GetService<ILogger<CatalogContextSeed>>();
new CatalogContextSeed()
.SeedAsync(context, env, settings, logger)
.Wait();
})
.MigrateDbContext<IntegrationEventLogContext>((_, __) => { });
Log.Information("Starting web host ({ApplicationContext})...", AppName);
host.Run();
return 0;
}
catch (Exception ex)
{
Log.Fatal(ex, "Program terminated unexpectedly ({ApplicationContext})!", AppName);
return 1;
}
finally
{
Log.CloseAndFlush();
}
}
There's an important caveat when applying migrations and seeding a database during container startup. Since the database server might not be available for whatever reason, you must handle retries while waiting for the server to be available. This retry logic is handled by the MigrateDbContext() extension method, as shown in the following code:
public static IWebHost MigrateDbContext<TContext>(
this IWebHost host,
Action<TContext,
IServiceProvider> seeder)
where TContext : DbContext
{
var underK8s = host.IsInKubernetes();
using (var scope = host.Services.CreateScope())
{
var services = scope.ServiceProvider;
var logger = services.GetRequiredService<ILogger<TContext>>();
var context = services.GetService<TContext>();
try
{
logger.LogInformation("Migrating database associated with context {DbContextName}", typeof(TContext).Name);
if (underK8s)
{
InvokeSeeder(seeder, context, services);
}
else
{
var retry = Policy.Handle<SqlException>()
.WaitAndRetry(new TimeSpan[]
{
TimeSpan.FromSeconds(3),
TimeSpan.FromSeconds(5),
TimeSpan.FromSeconds(8),
});
//if the sql server container is not created on run docker compose this
//migration can't fail for network related exception. The retry options for DbContext only
//apply to transient exceptions
// Note that this is NOT applied when running some orchestrators (let the orchestrator to recreate the failing service)
retry.Execute(() => InvokeSeeder(seeder, context, services));
}
logger.LogInformation("Migrated database associated with context {DbContextName}", typeof(TContext).Name);
}
catch (Exception ex)
{
logger.LogError(ex, "An error occurred while migrating the database used on context {DbContextName}", typeof(TContext).Name);
if (underK8s)
{
throw; // Rethrow under k8s because we rely on k8s to re-run the pod
}
}
}
return host;
}
The following code in the custom CatalogContextSeed class populates the data.
public class CatalogContextSeed
{
public static async Task SeedAsync(IApplicationBuilder applicationBuilder)
{
var context = (CatalogContext)applicationBuilder
.ApplicationServices.GetService(typeof(CatalogContext));
using (context)
{
context.Database.Migrate();
if (!context.CatalogBrands.Any())
{
context.CatalogBrands.AddRange(
GetPreconfiguredCatalogBrands());
await context.SaveChangesAsync();
}
if (!context.CatalogTypes.Any())
{
context.CatalogTypes.AddRange(
GetPreconfiguredCatalogTypes());
await context.SaveChangesAsync();
}
}
}
static IEnumerable<CatalogBrand> GetPreconfiguredCatalogBrands()
{
return new List<CatalogBrand>()
{
new CatalogBrand() { Brand = "Azure"},
new CatalogBrand() { Brand = ".NET" },
new CatalogBrand() { Brand = "Visual Studio" },
new CatalogBrand() { Brand = "SQL Server" }
};
}
static IEnumerable<CatalogType> GetPreconfiguredCatalogTypes()
{
return new List<CatalogType>()
{
new CatalogType() { Type = "Mug"},
new CatalogType() { Type = "T-Shirt" },
new CatalogType() { Type = "Backpack" },
new CatalogType() { Type = "USB Memory Stick" }
};
}
}
When you run integration tests, having a way to generate data consistent with your integration tests is useful. Being able to create everything from scratch, including an instance of SQL Server running on a container, is great for test environments.
EF Core InMemory database versus SQL Server running as a container
Another good choice when running tests is to use the Entity Framework InMemory database provider. You can specify that configuration in the ConfigureServices method of the Startup class in your Web API project:
public class Startup
{
// Other Startup code ...
public void ConfigureServices(IServiceCollection services)
{
services.AddSingleton<IConfiguration>(Configuration);
// DbContext using an InMemory database provider
services.AddDbContext<CatalogContext>(opt => opt.UseInMemoryDatabase());
//(Alternative: DbContext using a SQL Server provider
//services.AddDbContext<CatalogContext>(c =>
//{
// c.UseSqlServer(Configuration["ConnectionString"]);
//
//});
}
// Other Startup code ...
}
There is an important catch, though. The in-memory database does not support many constraints that are specific to a particular database. For instance, you might add a unique index on a column in your EF Core model and write a test against your in-memory database to check that it does not let you add a duplicate value. But when you are using the in-memory database, you cannot handle unique indexes on a column. Therefore, the in-memory database does not behave exactly the same as a real SQL Server database—it does not emulate database-specific constraints.
Even so, an in-memory database is still useful for testing and prototyping. But if you want to create accurate integration tests that take into account the behavior of a specific database implementation, you need to use a real database like SQL Server. For that purpose, running SQL Server in a container is a great choice and more accurate than the EF Core InMemory database provider.
Using a Redis cache service running in a container
You can run Redis on a container, especially for development and testing and for proof-of-concept scenarios. This scenario is convenient, because you can have all your dependencies running on containers—not just for your local development machines, but for your testing environments in your CI/CD pipelines.
However, when you run Redis in production, it is better to look for a high-availability solution like Redis Microsoft Azure, which runs as a PaaS (Platform as a Service). In your code, you just need to change your connection strings.
Redis provides a Docker image with Redis. That image is available from Docker Hub at this URL:
https://hub.docker.com/_/redis/
You can directly run a Docker Redis container by executing the following Docker CLI command in your command prompt:
docker run --name some-redis -d redis
The Redis image includes expose:6379 (the port used by Redis), so standard container linking will make it automatically available to the linked containers.
In eShopOnContainers, the basket-api microservice uses a Redis cache running as a container. That basketdata container is defined as part of the multi-container docker-compose.yml file, as shown in the following example:
#docker-compose.yml file
#...
basketdata:
image: redis
expose:
- "6379"
This code in the docker-compose.yml defines a container named basketdata based on the redis image and publishing the port 6379 internally. This configuration means that it will only be accessible from other containers running within the Docker host.
Finally, in the docker-compose.override.yml file, the basket-api microservice for the eShopOnContainers sample defines the connection string to use for that Redis container:
basket-api:
environment:
# Other data ...
- ConnectionString=basketdata
- EventBusConnection=rabbitmq
As mentioned before, the name of the microservice basketdata is resolved by Docker's internal network DNS.
