What strategy/technology should I use for this kind of replication?

I a开发者_如何学Pythonm currently facing one problem which not yet figure out good solution, so hope to get some advice from you all.

My Problem as in the picture

Core Database is where all the clients connect to for managing live data which is really really big and busy all the time.

Feature Database is not used so often but it need some part of live data (maybe 5%) from the Core Database, But the request task to this server will take longer time and consume much resource.

What is my current solution:

1. I used database replication between Core Database & Feature Database, it works fine. But the problem is that I waste a lot of disk space to store unwanted data. (Filtering while replicate data is not work with my databases schema)

2. Using queueing system will not make data live on time as there are many request to Core Database.

Please suggest some idea if you have met this?

Thanks,

Pang

What you define is a classic data integration task. You can use any data integration tool to extract data from your core database and load into featured database. You can schedule your data integration jobs from real-time to any time-frame.

I used Talend in my mid-size (10GB) semi-scientific PostgreSQL database integration project. It worked beautifully.

You can also try SQL Server Integration Services (SSIS). This tool is very powerful as well. It works with all top-notch RDBMSs.

If all you're worrying about is disk space, I would stick with the solution you have right now. 100GB of disk space is less than a dollar, these days - for that money, you can't really afford to bring a new solution into the system.

Logically, there's also a case to be made for keeping the filtering in the same application - keeping the responsibility for knowing which records are relevant inside the app itself, rather than in some mysterious integration layer will reduce overall solution complexity. Only accept the additional complexity of a special integration layer if you really need to.