[Twisted-Python] Scalability of an rss-aggregator

Fri Apr 2 00:12:41 MST 2004

Andrew Bennetts wrote:

>But the traceback you posted from the deferred chain was from the guts of
>feedparser...
>  
>
Looking at strace output it seems more like Twisted-related... Here is 
the output:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
gettimeofday({1080889267, 866232}, NULL) = 0
close(313)                              = 0
gettimeofday({1080889267, 866410}, NULL) = 0
select(726, [4 13 17 18 23 26 28 31 39 40 42 44 45 47 49 50 51 52 55 58 
60 63 66 67 69 73 74 77 78 80 81 82 86 89 92 98 99 101 104 108 109 110 
111 112 115 116 122 124 127 128 130 131 132 139 142 145 147 149 153 157 
158 162 164 171 172 173 176 178 180 182 183 185 187 188 190 191 192 193 
194 195 196 225 226 227 228 230 232 234 237 238 241 246 248 249 250 251 
253 254 256 257 258 260 262 263 264 267 268 269 271 273 274 275 277 278 
281 282 284 287 294 296 297 298 299 301 303 304 305 306 309 310 312 314 
316 317 321 322 325 327 329 330 331 332 334 335 336 339 340 343 345 346 
348 349 350 351 356 359 361 364 365 366 367 368 371 372 373 374 375 378 
379 380 383 385 386 387 390 393 396 397 399 400 401 405 406 407 408 410 
414 415 417 422 427 428 430 431 432 436 438 439 440 442 443 445 447 448 
450 451 452 453 455 456 458 459 462 464 467 468 476 477 479 480 481 482 
486 488 489 493 494 496 497 498 503 505 506 507 508 510 511 513 514 515 
517 520 523 525 527 528 529 534 535 537 538 539 544 547 550 553 554 556 
558 559 560 561 565 566 569 572 574 575 577 579 580 586 587 588 589 590 
592 593 594 597 598 600 601 602 603 604 605 606 609 610 611 612 613 615 
620 622 623 628 630 631 633 634 635 637 640 643 645 651 654 655 659 660 
664 665 666 667 670 671 675 676 677 678 681 683 684 687 715 725], [], 
[], NULL) = 1 (in [447])
recv(447, 0xa3cc9e4, 65536, 0)          = -1 ECONNRESET (Connection 
reset by peer)
shutdown(447, 2 /* send and receive */) = -1 ENOTCONN (Transport 
endpoint is not connected)
write(1, "parsing...\n", 11parsing...
)            = 11
futex(0x8067858, FUTEX_WAKE, 1)         = 0
futex(0x8067858, FUTEX_WAKE, 1)         = 0
futex(0x8067858, FUTEX_WAKE, 1)         = 0
write(1, "parsed feed\n", 12parsed feed
)           = 12
write(1, "Memoizing http://weblogs.asp.net"..., 53Memoizing 
http://weblogs.asp.net/JohanL/rss.aspx ...
) = 53
gettimeofday({1080889267, 968545}, NULL) = 0
write(1, "--------------------\n", 21--------------------
)  = 21
write(1, "finished retrieving\n", 20finished retrieving
)   = 20
write(1, "Feed Version: \n", 15Feed Version:
)        = 15
write(1, "--------------------\n", 21--------------------
)  = 21
write(1, "Closing connection number 404..."..., 33Closing connection 
number 404...
) = 33
write(1, "=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-"..., 
41=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
) = 41
write(1, "405 730\n", 8405 730
)                = 8
gettimeofday({1080889267, 974236}, NULL) = 0
close(447)                              = 0
gettimeofday({1080889267, 974428}, NULL) = 0
select(726, [4 13 17 18 23 26 28 31 39 40 42 44 45 47 49 50 51 52 55 58 
60 63 66 67 69 73 74 77 78 80 81 82 86 89 92 98 99 101 104 108 109 110 
111 112 115 116 122 124 127 128 130 131 132 139 142 145 147 149 153 157 
158 162 164 171 172 173 176 178 180 182 183 185 187 188 190 191 192 193 
194 195 196 225 226 227 228 230 232 234 237 238 241 246 248 249 250 251 
253 254 256 257 258 260 262 263 264 267 268 269 271 273 274 275 277 278 
281 282 284 287 294 296 297 298 299 301 303 304 305 306 309 310 312 314 
316 317 321 322 325 327 329 330 331 332 334 335 336 339 340 343 345 346 
348 349 350 351 356 359 361 364 365 366 367 368 371 372 373 374 375 378 
379 380 383 385 386 387 390 393 396 397 399 400 401 405 406 407 408 410 
414 415 417 422 427 428 430 431 432 436 438 439 440 442 443 445 448 450 
451 452 453 455 456 458 459 462 464 467 468 476 477 479 480 481 482 486 
488 489 493 494 496 497 498 503 505 506 507 508 510 511 513 514 515 517 
520 523 525 527 528 529 534 535 537 538 539 544 547 550 553 554 556 558 
559 560 561 565 566 569 572 574 575 577 579 580 586 587 588 589 590 592 
593 594 597 598 600 601 602 603 604 605 606 609 610 611 612 613 615 620 
622 623 628 630 631 633 634 635 637 640 643 645 651 654 655 659 660 664 
665 666 667 670 671 675 676 677 678 681 683 684 687 715 725], [], [], NULL

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Here it lock, but now, since I'm using strace, it won't restart after 
Ctrl+C.

As I said before... The download of _ALL_ feeds has already finished 
when twisted locks,

-- 
Valentino Volonghi aka Dialtone
Linux User #310274, Gentoo Proud User
X Python Newsreader developer
http://sourceforge.net/projects/xpn/