I was looking at Spring Batch Framework, and was trying to find a .NET compatible. For those of you who do know heard of it, “Spring Batch provides a technical framework and programming model to support long-running processes that perform a given set of tasks repetitively. A typical batch program generally reads a large number of records from a database, file, or queue, processes the data in some fashion, and then writes back data in a modified form. Spring Batch automates this basic batch iteration, providing the capability to process similar transactions as a set, typically in an offline environment without any user interaction.”
Batch processing is definitely unavoidable in large enterprises. There are lots of web application frameworks for Java and .NET, and new ones are emerging now and then. Interestingly, no many are developing batch framework for either .NET or Java. The only one I knew of for Java is Spring Batch, and for .NET, there is none that I know of.
I have used SQL Server Integration Service before in one of the projects to do some batch processing to copy data from flat file/Oracle database to SQL Server database. SQL Server Integration Service provide a GUI to do the normal ETL. However, it is not free, and also when I tried to benchmark it, the results are not really satisfactory.
I have developed then some batch processing routines using C#, and as a comparison with Spring Batch, certain features are the same. Maybe it will be interesting to develop a open source batch framework.
- Commit batch process periodically
- Concurrent batch processing: parallel processing of a job
Staged, enterprise message-driven processing
- Massively parallel batch processing
- Manual or scheduled restart after failure
- Sequential processing of dependent steps (with extensions to workflow-driven batches)
- Partial processing: skip records (e.g. on rollback)
- Whole-batch transaction: for cases with a simple enough data model or a small batch size
If anyone of you know of any open source .NET batch framework, please let me know.