In a MVC patterned framework where would a screen-scraping module be located?

In a MVC patterned framework where would a screen-scraping module most logically be located? In the model or the controller? Or is 开发者_运维百科it completely outside of this pattern?

You can call it as you might a model if you design it to behave like one. Then it can be easily used within a controller:

def update
  @company = Company.find(params[:id])

  @scraper = Scraper.find(:page => some_url, :method => :rip)

  @scraper.product_details.each do |params|
    @company.products.create(params)
  end
end

It is always convenient when you design the output format of your scraper to be compatible with some other object, such as the column mapping of your persistent storage.

It's often handy to have "model-like" objects. In this case, you can think of a scraper as an interface to another web site instead of a database, and in that respect it is not unlike ActiveResource. There's no rule that a model has to be built off of ActiveRecord.

In a sense, the screen-scraper is the user of the application -- I'd think it would be a piece outside the MVC that interacts with the controller, just like a web page sends information to the controller. This would make it easy to put a web page over it if the interface changes.

I presume the scraper is only used to collect data.

if the data is scraped and stored, this process does not simply drop into either M V or C. You could run this scrapping process as a cron job on intervals you define, store the data and then define models to access the data.

i wouldn't recommend calling the scrapping process every-time the controller requests the model, unless you are caching the scrap request.

EDIT:

the cron could be a controller, but not publicly callable.

It depends on how you are planning on implementing the scraper. If you are going to have a UI where you click a button to scrape a screen then it is going to be in all three (M, V, and C). If it is going to be a background process (like mentioned before) it should be in M and C.