Manhattan has three basic divisions, "uptown," "midtown," and "downtown." The financial district is contained within the geographic area of "downtown" (which starts at the Battery and has a nebulous northern border somewhere between the Village and 34th St).
The "Financial District" is not really "contained" in downtown anymore. The only major players left: Goldman, the NYSE, the AMEX. And Chase still has a building. The other major financial players have moved to midtown on Park Avenue, uptown from Grand Central and the Helmsley Building. For a while, they were building office parks in New Jersey. And what's taking their place? Residential. Yeah, those old office buildings have turned into luxury condos and co-ops.
For me, the upper limit of "Downtown" is the City Hall and Park Row where J&R used to be. Anything North of City Hall Park isn't proper Downtown. It's lower Chinatown, around Columbus Park where the old people do Tai Chi, whereas on the west side is TriBeCa.
Canal Street is the major dividing line running east-west, and hooks up with the Manhattan Bridge. North of that, you have more Chinatown, what's left of Little Italy, and SoHo to the West, and the Lower East Side to the East where you used to be able to get a cheap roach-filled apartment. Then you reach Houston Street, another major dividing line - North of it is Greenwich Village to the West, East Village to the East. The numbered streets start (Houston Street is essentially "0" street). It's all Village until you reach 14th Street.
It's kinda Midtown from there, 'cause it's no longer Village and damn-well ain't Downtown. But really it's Chelsea, Gramercy and Stuyvesant. Midtown, really, isn't until you get past Flat Iron to Nomad, Kips Bay and the Empire State Building at 34th Street. Then it's really Midtown, stretching up Northward until 59th Street which marks the lower end of Central Park... dividing the City between the Upper East Side to the East, and the Upper West Side to the, you get the idea. Get past the Park, you're in Harlem. North of that, Washington Heights and the GW Bridge to Chris Christie-land. Once you reach Inwood, there's nowhere else to go except across the Harlem River to the Bronx.
The town so nice, they named it twice. What I'd do for a slice.