 | | |
|
pfoley
User
Aug 21, 2002, 10:26 PM
Post #1 of 10
(4314 views)
Shortcut
|
|
import listings into RM
|
Can't Post
|
|
This has been mentioned somewhat in a couple of posts, but I'll expand on it a bit here.
Thanks to Broker Reciprocity, I need to important a batch of listings each night (and delete old ones). I was thinking of hacking listing.dat.cgi - obviously a brittle solution, but it seems to be the best I can do if I want this done within a couple of weeks. Am I wrong - is there a better solution that can be implemented quickly?
My basic strategy is to add a field that identifies the datasource and another for that datasource's mls number for the property. Each time I update a datasource, I'd calculate the properties that needed to be deleted (by comparing the existing properties in listing.dat.cgi with the import), then delete those rows in listing.data.cgi, then add the new rows and modify the existing ones. I'd want to be mindful of leaving the key fields alone, so that properties don't get renamed all the time.
Is this a sober approach? Is there a better way? I can't wait until the mySQL version is done!
If this sounds workable, I will ask some follow-up questions.
Thanks,
Patrick
|
|
| |
|
Dave
Staff
/ Moderator
Aug 26, 2002, 4:30 PM
Post #2 of 10
(4285 views)
Shortcut
|
|
Re: [pfoley] import listings into RM
[In reply to]
|
Can't Post
|
|
Sorry for the delay in getting back to you. Every now and then the forum seems to get ahead of us! :)
I'm not sure the best way to keep the listing file up to date with your import data but we can give you some skeleton code that loops over the data and inserts records. Then you can use that to check what's in the data file and add listings as needed. Let me know if that would help and I'll dig up the code.
Dave Edis - Senior Developer
interactivetools.com
|
|
| |
|
pfoley
User
Aug 26, 2002, 7:07 PM
Post #3 of 10
(4279 views)
Shortcut
|
|
Re: [Dave] import listings into RM
[In reply to]
|
Can't Post
|
|
That would be awesome! Thanks!
|
|
| |
|
Dave
Staff
/ Moderator
Aug 28, 2002, 10:54 AM
Post #4 of 10
(4262 views)
Shortcut
|
|
Re: [pfoley] import listings into RM
[In reply to]
|
Can't Post
|
|
Patrick,
I've attached a zip with some example programs and a perl library for accessing RM's data files. There's no documentation aside from the comments and examples in the code, but it should make the job much much easier than starting from scratch.
Hope that helps, let me know how it goes! : )
Dave Edis - Senior Developer
interactivetools.com
|
|
Attachments:
|
rm_developer_code_1.00.zip
(9.97 KB)
|
|
| |
|
pfoley
User
Aug 28, 2002, 12:38 PM
Post #5 of 10
(4257 views)
Shortcut
|
|
Re: [Dave] import listings into RM
[In reply to]
|
Can't Post
|
|
This is great - it will save me a bunch of time. Thanks!
|
|
| |
|
pfoley
User
Oct 17, 2002, 1:52 PM
Post #6 of 10
(3701 views)
Shortcut
|
|
Re: [pfoley] import listings into RM
[In reply to]
|
Can't Post
|
|
Several people have asked me about this - I'm just now finally getting to this work. I'll give an update when I finish.
Patrick
|
|
| |
|
pfoley
User
Oct 17, 2002, 1:58 PM
Post #7 of 10
(3700 views)
Shortcut
|
|
Re: [pfoley] import listings into RM
[In reply to]
|
Can't Post
|
|
Incidentally, the test programs work great - I urge anybody to try them. However, they assume you have a decent understanding of Perl.
One request for interactivetools if you have time ... My initial testing of rm-db.cgi leads me to believe that it will fill my needs with no modification - a wonderful thing to be sure. However, I am still publishing pages (listings and homepages) using http, which is somewhat cumbersome. Do you have a similar little file that handles the page publishing functions? If not, that would be high on my list of requests for future additions to your "programmatic interfaces" into RM.
Thanks again - this is going to be a breeze.
(This post was edited by pfoley on Oct 17, 2002, 2:09 PM)
|
|
| |
|
Miky
New User
Nov 7, 2002, 4:39 PM
Post #8 of 10
(3595 views)
Shortcut
|
|
Re: [pfoley] import listings into RM
[In reply to]
|
Can't Post
|
|
I would like to know a bit more about how to do the uploading or maybe someone out there knows of a good programer that could do some of the work?
Thanks
|
|
| |
|
pfoley
User
Nov 15, 2002, 8:32 AM
Post #9 of 10
(3476 views)
Shortcut
|
|
Re: [Miky] import listings into RM
[In reply to]
|
Can't Post
|
|
A lot of people have been interested in this topic. I implemented an import for one MLS (GRAR) for www.michaeljmay.com. I still have to import another MLS into the same site.
If you email me directly (see www.aptica.com), I will discuss my terms for doing custom work. I warn you, however, that I am not cheap.
Here are some pointers if you are going to try to do this yourself.
1. To do this project requires that you be at least an intermediate Perl programmer. It will be somewhat of a challenge for an intermediate programmer. It is pretty straightforward for an advanced programmer (but nothing is ever "easy", is it?).
2. You will have to figure out how to get the IDX listings from your MLS system - each one does it differently. There is no widely used standard. Typically, this is a big text file with one record per line and a delimiter between fields in each record (| is a common delimiter). You will have to figure out how to parse this file - meaning how to get each record and each field within each record. You likely won't use every field, but you will have to know how to get it.
3. You will have to make judgement calls to match the imported fields to your RM fields. For example, if your RM implementation uses Lake Front, Lake View, and Lake Access, but your import has Lake Frontage and Riperian Rights, what do you do? There is no set answer. You have to make a judgement.
4. You have to figure out how to get each import record into RM. The sample code uploaded by the kind folks at Interactive Tools was a huge help here (see the attachment earlier in this thread). I ended up modifying it for my own purposes, but it was a lot easier starting with their code. My modification involved writing all my imports at once, instead of closing the file after each record. This was simply a performance enhancement - it was not strictly necessary.
5. You will also have to figure out how to filter your import (limit the number of records). RM states a maximum number of listings of about 2000. That number will eventually rise dramatically if they add a mysql solution. Since my import has about 40,000 records, I am currently filtering on price, and I will probably add a zip code filter later on.
6. You have to figure out how to deal with "changing" records. If you do an update each day, then how do you handle the fact that imported property 57834 is already there? I handled this by simply deleting all imported records before I do an import each time. I noticed that I can assign an ID myself, instead of letting RM assign the ID. I use the MLS numbers from the import, which in my universe are guaranteed to be unique. Yours may not be - you'll have to figure out a different solution. In my solution any ID > 2000 was surely an import, so I deleted it before starting the import process.
NOTE THAT WE ARE USING UNDOCUMENTED FEATURES OF RM WHEN WE DO STUFF LIKE THIS! Interactive Tools gave us sample code, nothing more. They have not "published an interface". This makes solutions like mine "brittle". When RM comes up with a better way to store listings, my code will not work without modification. That's the nature of the beast.
7. You have to figure out how to publish pages and listings programmatically. It's a bit clunky, but right now, you have to use http to do it:
my $ua = new LWP::UserAgent;
$ua->agent("My Agent/0.1 " . $ua->agent);
my $req = new HTTP::Request GET => 'http://mysite.com/rm/exec/rm.cgi?login=1&id=RM&pw=mypass&setup_publish_listing_index=1';
my $res = $ua->request($req);
If the above doesn't make sense to you, then you are not yet ready to undertake this project on your own.
8. I didn't like showing imported listings on my "listings" page - I only wanted them to show up for searches. To accomplish this, I first delete all imported listings then publish the listing index, as shown above. Then I import all the listings and publish the listings themselves. THIS IS ALSO BRITTLE. I have limited (and fairly technical) RM users, so I can tell them "don't hit this button" (if they do a "publish listing index" manually, it would circumvent this system). You might not have that flexibility.
9. Before I write the file, I sort the records based on price. The posted sample code makes this straightforward.
10. You have to figure out how to trigger the import. The ideal solution is to schedule it to happen on a regular interval (daily). That was inconvenient in my case, so I gave my customer the ability to trigger the import by clicking a button on a private web page.
Hope this is useful to you. The good news is - if you procrastinate, RM will probably get more and more features that make this easier!
Patrick Foley
(This post was edited by pfoley on Nov 15, 2002, 12:34 PM)
|
|
| |
|
fraser_itools
Staff
Nov 15, 2002, 1:45 PM
Post #10 of 10
(3467 views)
Shortcut
|
|
Re: [pfoley] import listings into RM
[In reply to]
|
Can't Post
|
|
Patrick,
This is a fantastic blow-by-blow of your method to import records into Realty Manager. Who knows, you may very well create a business providing this service. :-)
You explained, perhaps better than we could, why creating an import/export system has been such a challenge. To create something that could match every single database out there would be largely impossible. Especially to make it usable by non-technical people.
It may have to remain in the domain of custom programming for quite a while.
Fraser Cain - Product Manager
interactivetools.com, inc.
Tel: (604)689-3347 - Fax: (604)689-3342 - Toll Free: 1(800)752-0455
Software for your Website - http://www.interactivetools.com/
|
|
| |
|
