Alper Akcan : ~/projects/libhthread

News

Projects

Documents

Contact

RSS Feed

Donations (SF)

 
libhthread
What is?

libhthread is a threads helper library and a thread detector with synchronization errors detection support.

project is hosted at GitHub. and a detailed documentation can be found at README.md

Documents

hthread

hthread is a thread error detector and helper library with synchronization errors detection support for c/c++ programs that use the pthreads.

  1. overview
  2. configuration
  3. error reports
  4. test cases
  5. usage example
  6. contact
  7. license

1. overview

hthread is a thread error detector and helper library with synchronization errors detection support for c/c++ programs that use the pthreads.

hthread is a lightweight thread error detector and helper library with synchronization error detection support, specifically designed for embedded systems.

main use case may include embedded systems where valgrind - drd, or helgrind support is not available.

and has benefits of:

  • has a negligible effect run-time speed
  • does not require any source code change
  • operating system and architecture independent
  • easy to use

can detect errors of:

  1. misuses of pthreads api
  2. lock ordering violation
  3. lock contention

1.1. misuses of pthreads api

hthread is able to detect and report following errors. although some of them are too obvious, early detection is much better than to deal with hard-to-find bugs.

  1. destroying an invalid mutex - fail-00.c, report
  2. locking an invalid mutex - fail-01.c, report
  3. unlocking an invalid mutex - fail-02.c, report
  4. locking an already locked mutex - fail-03.c, report
  5. unlocking an unheld mutex - fail-05.c, report
  6. destroying a locked mutex - fail-06.c, report
  7. destroying an invalid condition - fail-20.c, report
  8. signaling an invalid condition - fail-21.c, report
  9. broadcasting an invalid condition - fail-22.c, report
  10. [timed]waiting on an invalid condition - fail-23.c, report
  11. [timed]waiting on an invalid mutex - fail-24.c, report
  12. [timed]waiting on an unheld mutex - fail-25.c, report
  13. join invalid thread - fail-40.c, report
  14. detach invalid thread - fail-41.c, report
  15. unlocking mutex that was held by other thread - fail-42.c, report

1.2. lock ordering violation

hthread is able to detect inconsistent locking orders, it is very useful because they usually result to deadlocks. they may never be discovered during testing and lead to hard-to-find bugs.

hthread monitors locks/unlocks and stores them in separate tables for each thread. this allows hthread to build a global lock order table for running process. hthread checks for locking order violation against global order table for each lock request, and able to detect and report following errors.

  1. lock order violation in same thread - fail-04.c, report
  2. lock order violation while [timed]waiting on condition - fail-26.c, report
  3. lock order violation between threads - fail-43.c, report

1.3. lock contention

hthread able to understand and print report about lock contentions - a thread has to wait until requested lock is released. monitoring lock contentions is handy because, they usually cause unwanted delays or they may point to an undetected potential deadlock. hthread can report followings.

  1. waiting to lock a mutex more than allowed threshold - fail-60.c, report
  2. hold a mutex lock more than allowed threshold - fail-61.c, report

2. configuration

  1. compile-time options
  2. run-time options

2.1. compile-time options

hthread configuration parameters can be set using make flags, please check example section for demonstration.

  • HTHREAD_ENABLE_CALLSTACK

    default 1

    enable/disable reporting call trace information on error, useful but depends on libbdf, libdl, and backtrace function from glibc. may be disabled for toolchains which does not support backtracing.

  • HTHREAD_REPORT_CALLSTACK

    default 1

    dump callstack info (function call history) for error point.

  • HTHREAD_ASSERT_ON_ERROR

    default 1

    terminate the process on any pthreads api misuse and/or lock order violation.

  • HTHREAD_LOCK_THRESHOLD

    default 5000 miliseconds

    print report if lock is held longer than the specified time, in miliseconds.

  • HTHREAD_LOCK_THRESHOLD_ASSERT

    default 0

    terminate if lock is held longer than the specified time, in miliseconds.

  • HTHREAD_LOCK_TRY_THRESHOLD

    default 5000 miliseconds

    print report if locking operation takes longer than the specified time, in miliseconds.

  • HTHREAD_LOCK_TRY_THRESHOLD_ASSERT

    default 0

    terminate if locking operation takes longer than the specified time, in miliseconds.

2.2. run-time options

hthread reads configuration parameters from environment via getenv function call. one can either set/change environment variables in source code of monitored project via setenv function call, or set them globally in running shell using export function.

please check example section for demonstration.

  • hthread_report_callstack

    default 1

    dump callstack info (function call history) for error point.

  • hthread_assert_on_error

    default 1

    terminate the process on any pthreads api misuse and/or lock order violation.

  • hthread_lock_hreashold

    default 5000 miliseconds

    print report if lock is held longer than the specified time, in miliseconds.

  • hthread_lock_hreashold_assert

    default 0

    terminate if lock is held longer than the specified time, in miliseconds.

  • hthread_lock_try_threshold

    default 5000 miliseconds

    print report if locking operation takes longer than the specified time, in miliseconds.

  • hthread_lock_try_threshold_assert

    default 0

    terminate if locking operation takes longer than the specified time, in miliseconds.

3. error reports

  1. misuses of pthreads api
  2. lock ordering violation
  3. lock contention

3.1. misuses of pthreads api

  1. destroying an invalid mutex
  2. locking an invalid mutex
  3. unlocking an invalid mutex
  4. locking an already locked mutex
  5. unlocking an unheld mutex
  6. destroying a locked mutex
  7. destroying an invalid condition
  8. signaling an invalid condition
  9. broadcasting an invalid condition
  10. [timed]waiting on an invalid condition
  11. [timed]waiting on an invalid mutex
  12. [timed]waiting on an unheld mutex
  13. join invalid thread
  14. detach invalid thread
  15. unlocking mutex that was held by other thread

3.1.1. destroying an invalid mutex

example code: fail-00.c

# ./test/fail-00-debug 
(hthread:30786) new thread created: 'root-process (0x1427010)'
(hthread:30786)     at: (null) (null):0
(hthread:30786) mutex destroy with invalid mutex: '0x7fff1476d258'
(hthread:30786)     by: root-process (0x1427010)
(hthread:30786)     at: main fail-00.c:23
(hthread:30786)         0x40c12f: hthread.c (debug_dump_callstack:829)
(hthread:30786)         0x401b5f: fail-00.c (main:24)
(hthread:30786)         0x401bbd: (null) (_start:0)
fail-00-debug: hthread.c:1349: debug_mutex_del: Assertion `(mt != ((void *)0)) && "invalid mutex"' failed.

3.1.2. locking an invalid mutex

example code: fail-01.c

# ./test/fail-01-debug 
(hthread:30789) new thread created: 'root-process (0x1499010)'
(hthread:30789)     at: (null) (null):0
(hthread:30789) mutex lock with invalid mutex: '0x7fff394852f8'
(hthread:30789)     by: root-process (0x1499010)
(hthread:30789)     at: main fail-01.c:23
(hthread:30789)         0x408e83: hthread.c (debug_dump_callstack:829)
(hthread:30789)         0x40c2f4: hthread.c (debug_mutex_try_lock:1088)
(hthread:30789)         0x401b5f: fail-01.c (main:24)
(hthread:30789)         0x401bbd: (null) (_start:0)
fail-01-debug: hthread.c:946: debug_mutex_add_lock: Assertion `(mt != ((void *)0)) && "invalid mutex"' failed.

3.1.3. unlocking an invalid mutex

example code: fail-02.c

# ./test/fail-02-debug 
(hthread:30792) new thread created: 'root-process (0x1f9e010)'
(hthread:30792)     at: (null) (null):0
(hthread:30792) mutex unlock with invalid mutex '0x7ffffa879c38'
(hthread:30792)     by: root-process (0x1f9e010)
(hthread:30792)     at: main fail-02.c:23
(hthread:30792)         0x406a7c: hthread.c (debug_dump_callstack:829)
(hthread:30792)         0x40d619: hthread.c (hthread_mutex_unlock_actual_debug:603)
(hthread:30792)         0x401b5f: fail-02.c (main:24)
(hthread:30792)         0x401bbd: (null) (_start:0)
fail-02-debug: hthread.c:1226: debug_mutex_del_lock: Assertion `(mt != ((void *)0)) && "invalid mutex"' failed.

3.1.4. locking an already locked mutex

example code: fail-03.c

# ./test/fail-03-debug 
(hthread:30795) new thread created: 'root-process (0x1e670c0)'
(hthread:30795)     at: (null) (null):0
(hthread:30795) mutex lock with already held mutex: 'mutex(main fail-03.c:22) (0x1e67010)'
(hthread:30795)     by: root-process (0x1e670c0)
(hthread:30795)     at: main fail-03.c:32
(hthread:30795)         0x408ef2: hthread.c (debug_dump_callstack:829)
(hthread:30795)         0x40c4d4: hthread.c (debug_mutex_try_lock:1088)
(hthread:30795)         0x401cb8: fail-03.c (main:33)
(hthread:30795)         0x401da1: (null) (_start:0)
(hthread:30795)   previously acquired
(hthread:30795)     by: root-process (0x1e670c0)
(hthread:30795)     at: main fail-03.c:27
(hthread:30795)   created 'mutex(main fail-03.c:22) (0x1e67010)'
(hthread:30795)     at: main fail-03.c:22
fail-03-debug: hthread.c:963: debug_mutex_add_lock: Assertion `(mt == ((void *)0)) && "mutex is already locked"' failed.

3.1.5. unlocking an unheld mutex

example code: fail-05.c

# ./test/fail-05-debug 
(hthread:30801) new thread created: 'root-process (0x9200c0)'
(hthread:30801)     at: (null) (null):0
(hthread:30801) mutex unlock with unheld mutex: 'mutex(main fail-05.c:22) (0x920010)'
(hthread:30801)     by: root-process (0x9200c0)
(hthread:30801)     at: main fail-05.c:27
(hthread:30801)         0x406f94: hthread.c (debug_dump_callstack:829)
(hthread:30801)         0x40d7d9: hthread.c (hthread_mutex_unlock_actual_debug:603)
(hthread:30801)         0x401c95: fail-05.c (main:28)
(hthread:30801)         0x401d81: (null) (_start:0)
(hthread:30801)   created 'mutex(main fail-05.c:22) (0x920010)'
(hthread:30801)     at: main fail-05.c:22
fail-05-debug: hthread.c:1259: debug_mutex_del_lock: Assertion `(mtl != ((void *)0)) && "mutex is not locked"' failed.

3.1.6. destroying a locked mutex

example code: fail-06.c

# ./test/fail-06-debug 
(hthread:30804) new thread created: 'root-process (0x6420c0)'
(hthread:30804)     at: (null) (null):0
(hthread:30804) mutex destroy with currently locked mutex: '0x642010'
(hthread:30804)     by: root-process (0x6420c0)
(hthread:30804)     at: main fail-06.c:32
(hthread:30804)         0x40c21c: hthread.c (debug_dump_callstack:829)
(hthread:30804)         0x401cb8: fail-06.c (main:33)
(hthread:30804)         0x401d81: (null) (_start:0)
(hthread:30804)   lock observed
(hthread:30804)     by: root-process (0x6420c0)
(hthread:30804)     at: main fail-06.c:27
(hthread:30804)   created 'mutex(main fail-06.c:22) (0x642010)
(hthread:30804)     at: main fail-06.c:22
fail-06-debug: hthread.c:1366: debug_mutex_del: Assertion `(mt == ((void *)0)) && "invalid mutex"' failed.

3.1.7. destroying an invalid condition

example code: fail-20.c

# ./test/fail-20-debug 
(hthread:30807) new thread created: 'root-process (0x1829010)'
(hthread:30807)     at: (null) (null):0
(hthread:30807) cond destroy with invalid condition: '0x7fff491eb5a8'
(hthread:30807)     by: root-process (0x1829010)
(hthread:30807)     at: main fail-20.c:23
(hthread:30807)         0x40a413: hthread.c (debug_dump_callstack:829)
(hthread:30807)         0x401b5f: fail-20.c (main:24)
(hthread:30807)         0x401bbd: (null) (_start:0)
fail-20-debug: hthread.c:1436: debug_cond_del: Assertion `(cv != ((void *)0)) && "invalid condition"' failed.

3.1.8. signaling an invalid condition

example code: fail-21.c

# ./test/fail-21-debug 
(hthread:30810) new thread created: 'root-process (0x2019010)'
(hthread:30810)     at: (null) (null):0
(hthread:30810) cond signal with invalid condition: '0x7fffd0b4ec78'
(hthread:30810)     by: root-process (0x2019010)
(hthread:30810)     at: main fail-21.c:23
(hthread:30810)         0x405769: hthread.c (debug_dump_callstack:829)
(hthread:30810)         0x40a5a9: hthread.c (hthread_cond_signal_actual_debug:457)
(hthread:30810)         0x401b5f: fail-21.c (main:24)
(hthread:30810)         0x401c01: (null) (_start:0)
fail-21-debug: hthread.c:1469: debug_cond_check: Assertion `(cv != ((void *)0)) && "invalid condition"' failed.

3.1.9. broadcasting an invalid condition

example code: fail-22.c

# ./test/fail-22-debug 
(hthread:30813) new thread created: 'root-process (0x10b8010)'
(hthread:30813)     at: (null) (null):0
(hthread:30813) cond signal with invalid condition: '0x7fffafb97db8'
(hthread:30813)     by: root-process (0x10b8010)
(hthread:30813)     at: main fail-22.c:23
(hthread:30813)         0x405769: hthread.c (debug_dump_callstack:829)
(hthread:30813)         0x40a5d9: hthread.c (hthread_cond_broadcast_actual_debug:463)
(hthread:30813)         0x401b5f: fail-22.c (main:24)
(hthread:30813)         0x401c01: (null) (_start:0)
fail-22-debug: hthread.c:1469: debug_cond_check: Assertion `(cv != ((void *)0)) && "invalid condition"' failed.

3.1.10. [timed]waiting on an invalid condition

example code: fail-23.c

# ./test/fail-23-debug 
(hthread:30816) new thread created: 'root-process (0x10340c0)'
(hthread:30816)     at: (null) (null):0
(hthread:30816) cond timedwait with invalid condition: '0x7fffaab0c968'
(hthread:30816)     by: root-process (0x10340c0)
(hthread:30816)     at: main fail-23.c:43
(hthread:30816)         0x405a09: hthread.c (debug_dump_callstack:829)
(hthread:30816)         0x40a8db: hthread.c (hthread_cond_timedwait_tspec_actual_debug:495)
(hthread:30816)         0x401cf4: fail-23.c (main:43)
(hthread:30816)         0x401ea5: (null) (_start:0)
fail-23-debug: hthread.c:1469: debug_cond_check: Assertion `(cv != ((void *)0)) && "invalid condition"' failed.

3.1.11. [timed]waiting on an invalid mutex

example code: fail-24.c

# ./test/fail-24-debug 
(hthread:30819) new thread created: 'root-process (0x1e320c0)'
(hthread:30819)     at: (null) (null):0
(hthread:30819) cond timedwait with invalid mutex '0x7fff3e5d79d8'
(hthread:30819)     by: root-process (0x1e320c0)
(hthread:30819)     at: main fail-24.c:38
(hthread:30819)         0x406d1c: hthread.c (debug_dump_callstack:829)
(hthread:30819)         0x40a8b3: hthread.c (hthread_cond_timedwait_tspec_actual_debug:496)
(hthread:30819)         0x401cd1: fail-24.c (main:38)
(hthread:30819)         0x401e59: (null) (_start:0)
fail-24-debug: hthread.c:1226: debug_mutex_del_lock: Assertion `(mt != ((void *)0)) && "invalid mutex"' failed.

3.1.12. [timed]waiting on an unheld mutex

example code: fail-25.c

# ./test/fail-25-debug 
(hthread:30822) new thread created: 'root-process (0x23af0c0)'
(hthread:30822)     at: (null) (null):0
(hthread:30822) cond timedwait with unheld mutex: 'mutex(main fail-25.c:32) (0x23afa00)'
(hthread:30822)     by: root-process (0x23af0c0)
(hthread:30822)     at: main fail-25.c:52
(hthread:30822)         0x407184: hthread.c (debug_dump_callstack:829)
(hthread:30822)         0x40a9c3: hthread.c (hthread_cond_timedwait_tspec_actual_debug:496)
(hthread:30822)         0x401d72: fail-25.c (main:52)
(hthread:30822)         0x401f6d: (null) (_start:0)
(hthread:30822)   created 'mutex(main fail-25.c:32) (0x23afa00)'
(hthread:30822)     at: main fail-25.c:32
fail-25-debug: hthread.c:1259: debug_mutex_del_lock: Assertion `(mtl != ((void *)0)) && "mutex is not locked"' failed.

3.1.13. join invalid thread

example code: fail-40.c

# ./test/fail-40-debug 
(hthread:30828) new thread created: 'root-process (0xe20010)'
(hthread:30828)     at: (null) (null):0
(hthread:30828) join with invalid thread: '0x7fffecd84218'
(hthread:30828)     by: root-process (0xe20010)
(hthread:30828)     at: main fail-40.c:25
(hthread:30828)         0x404ff5: hthread.c (debug_dump_callstack:829)
(hthread:30828)         0x40daee: hthread.c (hthread_join_actual_debug:690)
(hthread:30828)         0x401b5f: fail-40.c (main:26)
(hthread:30828)         0x401bbd: (null) (_start:0)
fail-40-debug: hthread.c:349: hthread_check: Assertion `(th == thread) && "invalid thread"' failed.

3.1.14. detach invalid thread

example code: fail-41.c

# ./test/fail-41-debug 
(hthread:30831) new thread created: 'root-process (0x258a010)'
(hthread:30831)     at: (null) (null):0
(hthread:30831) detach with invalid thread: '0x7fff81b0b8d8'
(hthread:30831)     by: root-process (0x258a010)
(hthread:30831)     at: main fail-41.c:25
(hthread:30831)         0x404ff5: hthread.c (debug_dump_callstack:829)
(hthread:30831)         0x40db7e: hthread.c (hthread_detach_actual_debug:704)
(hthread:30831)         0x401b5f: fail-41.c (main:26)
(hthread:30831)         0x401bbd: (null) (_start:0)
fail-41-debug: hthread.c:349: hthread_check: Assertion `(th == thread) && "invalid thread"' failed.

3.1.15. unlocking mutex that was held by other thread

example code: fail-42.c

# ./test/fail-42-debug 
(hthread:30834) new thread created: 'root-process (0x1e560c0)'
(hthread:30834)     at: (null) (null):0
(hthread:30834) new thread created: 'thread(main fail-42.c:48) (0x1e56cd0)'
(hthread:30834)     at: main fail-42.c:48
(hthread:30834) mutex unlock with a mutex 'mutex(main fail-42.c:38) (0x1e56010)' currently hold by other thread
(hthread:30834)     by: thread(main fail-42.c:48) (0x1e56cd0)
(hthread:30834)     at: worker fail-42.c:23
(hthread:30834)         0x406d9d: hthread.c (debug_dump_callstack:829)
(hthread:30834)         0x40d929: hthread.c (hthread_mutex_unlock_actual_debug:603)
(hthread:30834)         0x401f4f: fail-42.c (worker:24)
(hthread:30834)         0x40da0a: hthread.c (thread_run:619)
(hthread:30834)         0x7f51f6de1e9a: pthread_create.c (start_thread:308)
(hthread:30834)   lock observed
(hthread:30834)     by: root-process (0x1e560c0)
(hthread:30834)     at: main fail-42.c:43
(hthread:30834)   created 'mutex(main fail-42.c:38) (0x1e56010)'
(hthread:30834)     at: main fail-42.c:38
fail-42-debug: hthread.c:1247: debug_mutex_del_lock: Assertion `(mtl == ((void *)0)) && "mutex is locked by other thread"' failed.

3.2. lock ordering violation

  1. lock order violation in same thread
  2. lock order violation while [timed]waiting on condition
  3. lock order violation between threads

3.2.1. lock order violation in same thread

example code: fail-04.c

# ./test/fail-04-debug 
(hthread:30798) new thread created: 'root-process (0x1d530c0)'
(hthread:30798)     at: (null) (null):0
(hthread:30798) mutex lock order 'mutex(main fail-04.c:26) (0x1d53b60)' before 'mutex(main fail-04.c:26) (0x1d53c10)' violated
(hthread:30798)   incorrect order is: acquisition of 'mutex(main fail-04.c:26) (0x1d53c10)'
(hthread:30798)       by: root-process (0x1d530c0)
(hthread:30798)       at: main fail-04.c:47
(hthread:30798)     followed by a later acquisition of 'mutex(main fail-04.c:26) (0x1d53b60)'
(hthread:30798)       by: root-process (0x1d530c0)
(hthread:30798)       at: main fail-04.c:47
(hthread:30798)           0x408b94: hthread.c (debug_dump_callstack:829)
(hthread:30798)           0x40c5c4: hthread.c (debug_mutex_try_lock:1088)
(hthread:30798)           0x401d20: fail-04.c (main:48)
(hthread:30798)           0x401e89: (null) (_start:0)
(hthread:30798)   required order is: acquisition of 'mutex(main fail-04.c:26) (0x1d53b60)'
(hthread:30798)       by: root-process (0x1d530c0)
(hthread:30798)       at: main fail-04.c:33
(hthread:30798)     followed by a later acquisition of 'mutex(main fail-04.c:26) (0x1d53c10)'
(hthread:30798)       by: root-process (0x1d530c0)
(hthread:30798)       at: main fail-04.c:33
(hthread:30798)   created 'mutex(main fail-04.c:26) (0x1d53c10)'
(hthread:30798)     at: main fail-04.c:26
(hthread:30798)   created 'mutex(main fail-04.c:26) (0x1d53b60)'
(hthread:30798)     at: main fail-04.c:26
fail-04-debug: hthread.c:1036: debug_mutex_add_lock: Assertion `(mto == ((void *)0)) && "lock order violation"' failed.

3.2.2. lock order violation while [timed]waiting on condition

example code: fail-26.c

# ./test/fail-26-debug 
(hthread:30825) new thread created: 'root-process (0x77b0c0)'
(hthread:30825)     at: (null) (null):0
(hthread:30825) cond wait order 'mutex(main fail-26.c:32) (0x77b010)' before 'mutex(main fail-26.c:32) (0x77ba00)' violated
(hthread:30825)   incorrect order is: acquisition of 'mutex(main fail-26.c:32) (0x77ba00)'
(hthread:30825)       by: root-process (0x77b0c0)
(hthread:30825)       at: main fail-26.c:48
(hthread:30825)     followed by a later acquisition of 'mutex(main fail-26.c:32) (0x77b010)'
(hthread:30825)       by: root-process (0x77b0c0)
(hthread:30825)       at: main fail-26.c:57
(hthread:30825)           0x408cb4: hthread.c (debug_dump_callstack:829)
(hthread:30825)           0x40aa3b: hthread.c (hthread_cond_timedwait_tspec_actual_debug:513)
(hthread:30825)           0x401d95: fail-26.c (main:57)
(hthread:30825)           0x401fb5: (null) (_start:0)
(hthread:30825)   required order is: acquisition of 'mutex(main fail-26.c:32) (0x77b010)'
(hthread:30825)       by: root-process (0x77b0c0)
(hthread:30825)       at: main fail-26.c:43
(hthread:30825)     followed by a later acquisition of 'mutex(main fail-26.c:32) (0x77ba00)'
(hthread:30825)       by: root-process (0x77b0c0)
(hthread:30825)       at: main fail-26.c:48
(hthread:30825)   created 'mutex(main fail-26.c:32) (0x77ba00)'
(hthread:30825)     at: main fail-26.c:32
(hthread:30825)   created 'mutex(main fail-26.c:32) (0x77b010)'
(hthread:30825)     at: main fail-26.c:32
fail-26-debug: hthread.c:1036: debug_mutex_add_lock: Assertion `(mto == ((void *)0)) && "lock order violation"' failed.

3.2.3. lock order violation between threads

example code: fail-43.c

# ./test/fail-43-debug 
(hthread:30838) new thread created: 'root-process (0x14ae0c0)'
(hthread:30838)     at: (null) (null):0
(hthread:30838) new thread created: 'thread(main fail-43.c:73) (0x14af0f0)'
(hthread:30838)     at: main fail-43.c:73
(hthread:30838) mutex lock order 'mutex(main fail-43.c:53) (0x14ae010)' before 'mutex(main fail-43.c:58) (0x14aea00)' violated
(hthread:30838)   incorrect order is: acquisition of 'mutex(main fail-43.c:58) (0x14aea00)'
(hthread:30838)       by: thread(main fail-43.c:73) (0x14af0f0)
(hthread:30838)       at: worker fail-43.c:23
(hthread:30838)     followed by a later acquisition of 'mutex(main fail-43.c:53) (0x14ae010)'
(hthread:30838)       by: thread(main fail-43.c:73) (0x14af0f0)
(hthread:30838)       at: worker fail-43.c:28
(hthread:30838)           0x408d24: hthread.c (debug_dump_callstack:829)
(hthread:30838)           0x40c754: hthread.c (debug_mutex_try_lock:1088)
(hthread:30838)           0x40206f: fail-43.c (worker:29)
(hthread:30838)           0x40db5a: hthread.c (thread_run:619)
(hthread:30838)           0x7f58cbd27e9a: pthread_create.c (start_thread:308)
(hthread:30838)   required order is: acquisition of 'mutex(main fail-43.c:53) (0x14ae010)'
(hthread:30838)       by: root-process (0x14ae0c0)
(hthread:30838)       at: main fail-43.c:63
(hthread:30838)     followed by a later acquisition of 'mutex(main fail-43.c:58) (0x14aea00)'
(hthread:30838)       by: root-process (0x14ae0c0)
(hthread:30838)       at: main fail-43.c:68
(hthread:30838)   created 'mutex(main fail-43.c:58) (0x14aea00)'
(hthread:30838)     at: main fail-43.c:58
(hthread:30838)   created 'mutex(main fail-43.c:53) (0x14ae010)'
(hthread:30838)     at: main fail-43.c:53
fail-43-debug: hthread.c:1036: debug_mutex_add_lock: Assertion `(mto == ((void *)0)) && "lock order violation"' failed.

3.3. lock contention

  1. waiting to lock a mutex more than allowed threshold
  2. hold a mutex lock more than allowed threshold

3.3.1. waiting to lock a mutex more than allowed threshold

example code: fail-60.c

lock try threshold is set to 1 seconds, and enabled terminating if waited more than threshold time.

# hthread_lock_try_threshold=1000 hthread_lock_try_threshold_assert=1 ./test/fail-60-debug 
(hthread:30880) new thread created: 'root-process (0x13670c0)'
(hthread:30880)     at: (null) (null):0
(hthread:30880) new thread created: 'thread(main fail-60.c:53) (0x1367cd0)'
(hthread:30880)     at: main fail-60.c:53
(hthread:30880) mutex lock still waiting for mutex: 'mutex(main fail-60.c:43) (0x1367010)'
(hthread:30880)     by: thread(main fail-60.c:53) (0x1367cd0)
(hthread:30880)     at: worker fail-60.c:23
(hthread:30880)         0x40d6e5: hthread.c (debug_dump_callstack:829)
(hthread:30880)         0x401fbf: fail-60.c (worker:24)
(hthread:30880)         0x40da8a: hthread.c (thread_run:619)
(hthread:30880)         0x7f5bf1a76e9a: pthread_create.c (start_thread:308)
(hthread:30880)   currently locked
(hthread:30880)     by: root-process (0x13670c0)
(hthread:30880)     at: main fail-60.c:48
(hthread:30880)   currently locked
(hthread:30880)     by: thread(main fail-60.c:53) (0x1367cd0)
(hthread:30880)     at: worker fail-60.c:23
(hthread:30880)   created 'mutex(main fail-60.c:43) (0x1367010)'
(hthread:30880)     at: main fail-60.c:43
fail-60-debug: hthread.c:1166: debug_mutex_try_lock: Assertion `(a == 0) && "mutex try lock threshold reached"' failed.

3.3.2. hold a mutex lock more than allowed threshold

example code: fail-61.c

lock hold threshold is set to 1 seconds, and enabled terminating if waited more than threshold time.

# hthread_lock_threshold=1000 hthread_lock_threshold_assert=1 ./test/fail-61-debug 
(hthread:30885) new thread created: 'root-process (0x15a20c0)'
(hthread:30885)     at: (null) (null):0
(hthread:30885) mutex unlock with a mutex 'mutex(main fail-61.c:22) (0x15a2010)' hold during 3001 ms
(hthread:30885)     by: root-process (0x15a20c0)
(hthread:30885)     at: main fail-61.c:33
(hthread:30885)         0x406bc6: hthread.c (debug_dump_callstack:829)
(hthread:30885)         0x40d849: hthread.c (hthread_mutex_unlock_actual_debug:603)
(hthread:30885)         0x401d06: fail-61.c (main:34)
(hthread:30885)         0x401ded: (null) (_start:0)
(hthread:30885)   lock observed
(hthread:30885)     by: root-process (0x15a20c0)
(hthread:30885)     at: main fail-61.c:27
(hthread:30885)   created 'mutex(main fail-61.c:22) (0x15a2010)'
(hthread:30885)     at: main fail-61.c:22
fail-61-debug: hthread.c:1286: debug_mutex_del_lock: Assertion `(a == 0) && "mutex lock threshold reached"' failed.

4. test cases

  1. mutex tests
  2. condition tests
  3. thread tests
  4. lock contention tests

4.1. mutex tests

test sucess fail
sucess-00.c
fail-00.c
init mutex
destroy mutex
destroy mutex
** invalid mutex **
sucess-01.c
fail-01.c
init mutex
lock mutex
unlock mutex
destroy mutex
lock mutex
** invalid mutex **
sucess-02.c
fail-02.c
init mutex
lock mutex
unlock mutex
destroy mutex
unlock mutex
** invalid mutex **
sucess-03.c
fail-03.c
init mutex
lock mutex
unlock mutex
destroy mutex
init mutex
lock mutex
lock mutex
** mutex is already locked **
sucess-04.c
fail-04.c
init mutexes
lock mutexes 0...n
unlock mutexes 0...n
lock mutexes 0...n
unlock mutexes 0...n
destroy mutexes
init mutexes
lock mutexes 0...n
unlock mutexes 0...n
lock mutexes n...0
** lock order violation **
sucess-05.c
fail-05.c
init mutex
lock mutex
unlock mutex
destroy mutex
init mutex
unlock mutex
** unlocking a unheld mutex **
sucess-06.c
fail-06.c
init mutex
lock mutex
unlock mutex
destroy mutex
init mutex
lock mutex
destroy mutex
** destroying a locked mutex **

4.2. condition tests

test sucess fail
sucess-20.c
fail-20.c
init condition
destroy condition
destroy condition
** invalid condition **
sucess-21.c
fail-21.c
init condition
signal condition
destroy condition
signal condition
** invalid condition **
sucess-22.c
fail-22.c
init condition
broadcast condition
destroy condition
broadcast condition
** invalid condition **
sucess-23.c
fail-23.c
init condition
init mutex
lock mutex
timed wait on condition, mutex
unlock mutex
destroy condition
destroy mutex
init mutex
lock mutex
timed wait on condition, mutex
** invalid condition **
sucess-24.c
fail-24.c
init condition
init mutex
lock mutex
timed wait on condition, mutex
unlock mutex
destroy condition
destroy mutex
init condition
timed wait on condition, mutex
** invalid mutex **
sucess-25.c
fail-25.c
init mutexes
init condition
lock mutex 0
timed wait on condition, mutex 0
unlock mutex 0
destroy condition
destroy mutexes
init mutexes
init condition
lock mutex 0
timed wait on condition, mutex 1
** mutex not locked **
sucess-26.c
fail-26.c
init mutexes
init condition
lock mutex 0
lock mutex 1
unlock mutex 1
timed wait on condition, mutex 0
unlock mutex 0
destroy condition
destroy mutexes
init mutexes
init condition
lock mutex 0
lock mutex 1
timed wait on condition, mutex 0
** lock order will be violated **

4.3. thread tests

test sucess fail
sucess-40.c
fail-40.c
create thread
join thread
join thread
** invalid thread **
sucess-41.c
fail-41.c
create thread
detach thread
detach thread
** invalid thread **
sucess-42.c
fail-42.c
main : init mutex
main : lock mutex
main : create thread
thread: lock mutex
thread: unlock mutex
main : unlock mutex
main : join thread
main : destroy mutex
main : init mutex
main : lock mutex
main : create thread
thread: unlock mutex
** unlocking mutex currently held **
** by other thread **
sucess-43.c
fail-43.c
main : init mutex 0
main : init mutex 1
main : lock mutex 0
main : lock mutex 1
main : create thread
thread: lock mutex 0
thread: lock mutex 1
thread: unlock mutex 0
thread: unlock mutex 1
main : unlock mutex 0
main : unlock mutex 1
main : join thread
main : destroy mutex
main : init mutex 0
main : init mutex 1
main : lock mutex 0
main : lock mutex 1
main : create thread
thread: lock mutex 1
thread: lock mutex 0
** lock order violation **

4.4. lock contention tests

test sucess fail
sucess-60.c
fail-60.c
hthread_lock_try_threshold = 1000
hthread_lock_try_threshold_assert = 0
main : init mutex
main : lock mutex
main : create thread
main : sleep 3
thread: lock mutex
** try lock threshold reached **
** still waiting on mutex **
thread: unlock mutex
main : unlock mutex
main : join thread
main : destroy mutex
hthread_lock_try_threshold = 1000
hthread_lock_try_threshold_assert = 1
main : init mutex
main : lock mutex
main : create thread
main : sleep 3
thread: lock mutex
** try lock threshold reached **
** still waiting on mutex **
** assert requested **
sucess-61.c
fail-61.c
hthread_lock_threshold = 1000
hthread_lock_threshold_assert = 0
init mutex
lock mutex
sleep 3
unlock mutex
** lock threshold reached **
destroy mutex
hthread_lock_threshold = 1000
hthread_lock_assert = 1
init mutex
lock mutex
sleep 3
unlock mutex
** lock threshold reached **
** assert requested **

5. usage example

using hthread is pretty simple, just clone libhthread and build;

  • add -include hthread.h -DHTHREAD_DEBUG=1 -g -O1 to target cflags
  • link with -lhthread -lrt if HTHREAD_ENABLE_CALLSTACK is 0 or
  • link with -lhthread -lrt -ldl -lbfd if HTHREAD_ENABLE_CALLSTACK is 1

compile libhthread with callstack support

# git clone git://github.com/anhanguera/libhthread.git
# cd libhthread
# HTHREAD_ENABLE_CALLSTACK=1 make

compile libhthread without callstack support

# git clone git://github.com/anhanguera/libhthread.git
# cd libhthread
# HTHREAD_ENABLE_CALLSTACK=0 make

let below is the source code - with double lock error - to be monitored:

1  #include <stdio.h>
2  #include <stdlib.h>
3  #include <unistd.h>
4  #include <pthread.h>
5  
6  int main (int argc, char *argv[])
7  {
8    int rc;
9    pthread_mutex_t m;
10   (void) argc;
11   (void) argv;
12   rc = pthread_mutex_init(&m, NULL);
13   if (rc != 0) {
14     fprintf(stderr, "pthread_mutex_init failed\n");
15     exit(-1);
16   }
17   rc = pthread_mutex_lock(&m);
18   if (rc != 0) {
19     fprintf(stderr, "pthread_mutex_lock failed\n");
20     exit(-1);
21   }
22   rc = pthread_mutex_lock(&m);
23   if (rc != 0) {
24     fprintf(stderr, "pthread_mutex_lock failed\n");
25     exit(-1);
26   }
27   rc = pthread_mutex_destroy(&m);
28   if (rc != 0) {
29     fprintf(stderr, "pthread_mutex_destroy failed\n");
30     exit(-1);
31   }
32   return 0;
32 }

compile and run as usual:

# gcc -o app main.c -lpthread
# ./app

application will not exit, because it is trying to lock a already locked mutex. now, enable monitoring with hthread:

# gcc -include src/hthread.h -DHTHREAD_DEBUG=1 -g -O1 -o app-debug main.c -Lsrc -lhthread -lrt -ldl -lbfd -lpthread
# LD_LIBRARY_PATH=src ./app-debug
(hthread:30967) new thread created: 'root-process (0x11850b0)'
(hthread:30967)     at: (null) (null):0
(hthread:30967) mutex lock with already held mutex: 'mutex(main main.c:12) (0x1185010)'
(hthread:30967)     by: root-process (0x11850b0)
(hthread:30967)     at: main main.c:22
(hthread:30967)         0x7f744a1d0d52: hthread.c (debug_dump_callstack:829)
(hthread:30967)         0x7f744a1d4334: hthread.c (debug_mutex_try_lock:1088)
(hthread:30967)         0x400986: main.c (main:23)
(hthread:30967)         0x4007e9: (null) (_start:0)
(hthread:30967)   previously acquired
(hthread:30967)     by: root-process (0x11850b0)
(hthread:30967)     at: main main.c:17
(hthread:30967)   created 'mutex(main main.c:12) (0x1185010)'
(hthread:30967)     at: main main.c:12
app-debug: hthread.c:963: debug_mutex_add_lock: Assertion `(mt == ((void *)0)) && "mutex is already locked"' failed.

hthread detected and reported the error: application was trying to lock an already locked mutex at line 22, which was previously locked at line 17, and was created at line 12. and terminated the process.

program termination on error can be disabled with hthread_assert_on_error configuration parameter

# hthread_assert_on_error=0 LD_LIBRARY_PATH=src ./app-debug
(hthread:32648) new thread created: 'root-process (0xab10b0)'
(hthread:32648)     at: (null) (null):0
(hthread:32648) mutex lock with already held mutex: 'mutex(main main.c:12) (0xab1010)'
(hthread:32648)     by: root-process (0xab10b0)
(hthread:32648)     at: main main.c:22
(hthread:32648)   previously acquired
(hthread:32648)     by: root-process (0xab10b0)
(hthread:32648)     at: main main.c:17
(hthread:32648)   created 'mutex(main main.c:12) (0xab1010)'
(hthread:32648)     at: main main.c:12
hthread::error: (mt == NULL) && "mutex is already locked" (debug_mutex_add_lock hthread.c:823)

this time hthread detected and reported the error, but not terminated the process.

6. contact

if you are using the software and/or have any questions, suggestions, etc. please contact with me at alper.akcan@gmail.com

7. license

Copyright (C) 2009-2013 Alper Akcan alper.akcan@gmail.com

This work is free. It comes without any warranty, to the extent permitted by applicable law. You can redistribute it and/or modify it under the terms of the Do What The Fuck You Want To Public License, Version 2, as published by Sam Hocevar. See the COPYING file for more details.

Downloads

File Name
        
Date
        
File Size

 
libhthread-git-snapshot.zip        unkown        unkown bytes

(CL) alper akcan
http://www.valgrind.org   hacker emblem   Valid HTML 4.01!   Viewable With Any Browser   [Valid Rss]   Open Source