Successful invasion of the maternal vascular system by trophoblast cells is a prerequisite for the establishment of a normal hemochorial placenta. Transforming growth factor-beta (TGFbeta) has been implicated in the regulation of trophoblast invasiveness into the uterus. Endoglin is a component of the TGFbeta receptor complex that binds beta1 and beta3 isoforms and is expressed at high levels on syncytiotrophoblast throughout pregnancy and is also transiently up-regulated on extravillous trophoblasts differentiating along the invasive pathway. We investigated the role of endoglin in a serum-free human villous explant culture system that allows the study of trophoblast outgrowth, migration, and invasion and mimics events occurring in anchoring villi during the first trimester of gestation. Addition to explant cultures from 5-8 weeks gestation of a monoclonal antibody to endoglin or of antisense endoglin oligonucleotides significantly stimulated trophoblast outgrowth and migration. These responses were specific, as incubation of explants with nonimmune IgG or sense and scrambled oligonucleotides had no effect. Antisense endoglin-induced trophoblast outgrowth and migration were accompanied by cell division of villous-associated trophoblasts within the proximal region of the forming column and by the characteristic switch in integrins observed in anchoring villi in situ. Treatment of villous explants with antibody and antisense oligonucleotides to endoglin also resulted in an increased fibronectin release into the culture medium. The stimulatory effect of antisense endoglin on fibronectin production was overcome by the addition of exogenous TGFbeta2, but not TGFbeta1 and -beta3. These findings suggest that endoglin expression in the transition from polarized to nonpolarized trophoblasts in anchoring villi is necessary for mediation of the inhibitory effect of TGFbeta1 and/or TGFbeta3 on trophoblast differentiation along the invasive pathway.