python - How to convert a DataFrame back to normal RDD in pyspark? - Stack Overflow
Join the Stack Overflow Community
Stack Overflow is a community of 6.9 million programmers, just like you, helping each other.
J it only takes a minute:
I need to use the
(rdd.)partitionBy(npartitions, custom_partitioner)
method that is not available on the DataFrame.
All of the DataFrame methods refer only to DataFrame results.
So then how to create an RDD from the DataFrame data?
Note: this is a change (in 1.3.0) from 1.2.0.
Update from the answer from @dpangmao: the method is .rdd. I was interested to understand if (a) it were public and (b) what are the performance implications.
Well (a) is yes and (b) - well you can see here that there are significant perf implications: a new RDD must be created
by invoking mapPartitions :
In dataframe.py (note the file name changed as well (was sql.py):
def rdd(self):
Return the content of the :class:`DataFrame` as an :class:`RDD`
of :class:`Row` s.
if not hasattr(self, '_lazy_rdd'):
jrdd = self._jdf.javaToPython()
rdd = RDD(jrdd, self.sql_ctx._sc, BatchedSerializer(PickleSerializer()))
schema = self.schema
def applySchema(it):
cls = _create_cls(schema)
return itertools.imap(cls, it)
self._lazy_rdd = rdd.mapPartitions(applySchema)
return self._lazy_rdd
22.6k174186
13.7k1380174
Use the method .rdd like this:
rdd = df.rdd
22.6k174186
@dapangmao's answer works, but it doesn't give the regular spark RDD, it returns a Row object. If you want to have the regular RDD format.
rdd = df.rdd.map(tuple)
rdd = df.rdd.map(list)
11.1k743102
Your Answer
Sign up or
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Post as a guest
By posting your answer, you agree to the
Not the answer you're looking for?
Browse other questions tagged
rev .25527
Stack Overflow works best with JavaScript enabledto back onto sth的意思
沪江词库精选to back onto sth是什么意思、英语短语。
中文释义： 背面靠近
电磁流体力学
电磁液压阀
喇叭天线，喇叭形天线
电磁式接地检测器
电磁传动装置
2017to back onto sth是什么意思由沪江网提供。一键安装桌面版，查词翻译快三倍
选择要添加的生词本
词霸Android版
牛津词典免费使用! 立刻扫码下载
牛津词典免费使用! 立刻扫码下载
创建新的生词本
i该生词本已经创建啦!
i不可以出现中文，英文，数字之外的符号哒!
i生词本名称长度不能大于24字符!
i请填写生词本名称!Denise Richards and girls return to normality after celebrating Charlie Sheen's birthday | Daily Mail Online
Back to school! Denise Richards and her girls return to normality days after celebrating Charlie Sheen's birthday
07:52 BST, 7 September 2016
09:56 BST, 7 September 2016
Just days ago they were enjoying a private jet birthday date with Charlie Sheen.But his ex-wife Denise Richards and their girls were brought back to reality with a bump as they headed to school in Malibu on Tuesday.The Starship Troopers actress and their little ones were spotted going back for lessons after they enjoyed a lunch date in the California celebrity enclave.
Back to normal: Denise Richards was spotted taking her girls to school in Malibu on TuesdayWild Things favourite Denise, 45, was looking good for her age in a black vest, skinny jeans and flip flops.Her oldest daughter Sam, 11, looked trendy in a white polo shirt, tartan skirt and trainers, while her 10-year-old sister Lola wore a navy shirt, tan skirt and white shoes.
RELATED ARTICLES
Share this article
The girls certainly did not look in as good a mood as they did on Monday, when they celebrated their father Charie's 51st birthday, his first since announcing he is HIV positive.
Missing from the family get together were Charlie's seven-year-old twin boys Bob and Max, with third ex-wife Brooke Mueller, and 30-year-old Cassandra from his relationship with his high school sweetheart Paula Speert.
Vested interest: Well-weathered Denise was turning heads in her dark top
Food for thought: She had treated her girls to lunch before their afternoon lessons
Pretty fly: They enjoyed a private jet birthday date with Charlie Sheen at the weekendThe happy family snap comes after it was reported last week that Denise and Brooke, 39, had agreed to reduced child support payments.The celebrity is reportedly struggling with financial problems.Both women reportedly agreed to around $25,000 a month, nearly half of the $55,000 a month they had been previously been receiving to cover child-care costs.Charlie was the highest paid actor on television in 2010 when he commanded $1.8 million per episode of Two And A Half Men, before his contract was terminated amid reports of drug and alcohol abuse and a public meltdown.
Happier times: Charlie and his former wife at the premiere of Undercover Brother in 2002
Share or comment on this article
MOST READ NEWS
The comments below have not been moderated.
We are no longer accepting comments on this article.
More top stories
DON'T MISS
MORE DON'T MISS
SHARE PICTURE
Copy link to paste in your message